Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions tensorrt_llm/_torch/modeling_v2/models/inkling2/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: Apache-2.0 -->

# Inkling 2

`Inkling2ForConditionalGeneration` (`thinkingmachines/Inkling-2-Small`): a 48-layer MoE text decoder
(hidden 4608, 384 routed experts, top 8) with a vision tower. The family implements the text model.

## What the architecture needs

* **Attention without rotary embeddings.** Position enters through a relative bias each token mixes
per head from 16 bias-vs-distance profiles (`rel_logits_proj`), over distances below 1024. 40
sliding-window layers attend the last 1024 tokens; 8 global layers (5, 11, ..., 47) attend the
whole prefix and scale q and the relative states by `tau = 1 + 0.1 ln(max((p + 1) / 8192, 1))`.
Each token and head has a sink logit, and the output is gated by `sigmoid(gate)`. q, k and the
relative states are L2-clipped; v is RMS-normalized without a weight.
* **Short convolutions.** Width-4 causal convolutions with a residual run on k and v (dynamic
per-token taps on the global layers) and on each sublayer's output. Their windows are
per-request state.
* **Block attention residuals.** Every 6 layers the residual stream closes a block; each
sublayer reads a learned softmax mix of the normalized blocks and the current partial block.
* **Experts.** fp32 sigmoid router with a selection bias; SiTU activation
(`4 tanh(g / 4) sigmoid(g) * 25 tanh(u / 25)`). The checkpoint stores layers 0-39 in NVFP4 (with
per-token 4/6 NVFP4 activations) and layers 40-47 in MXFP8.

## Layout

```text
inkling2/
├── config.py # InklingDims: the decoder's shape and per-layer expert formats
├── kernels/ # Triton: short convolution, paged attention with the relative bias
├── ops/ # OpWrappers: conv, attention prologue and core, residual mix,
│ # router, NVFP4 experts (FlashInfer CuTe DSL), MXFP8 experts
└── inkling_2_small_nvfp4__sm_103__tep4/
├── weights.py # weight table: TP4 heads, EP4 experts, kernel layouts
├── modeling.py # the 48-layer forward and the engine shell
└── cache.py # KVCacheManagerV2 with the convolution windows
```

## Target `inkling_2_small_nvfp4__sm_103__tep4`

The released checkpoint on one GB300 tray: attention TP4 (12 query and 2 KV heads per rank), MoE
EP4 (96 experts per rank), BF16 KV cache. The baseline runs eagerly, without CUDA graphs, the
overlap scheduler, chunked prefill, KV cache block reuse or speculative decoding.

The family is not routed yet: these are the operators and the target's components; the route that
serves the target through `LLM()` and `trtllm-serve` follows separately.

The NVFP4 experts run FlashInfer's CuTe DSL fused MoE with SiTU and per-token 4/6 activations. The
MXFP8 experts run a grouped Triton MoE that expands the E4M3 weights to bf16 in registers (exactly)
and accumulates in fp32.
3 changes: 3 additions & 0 deletions tensorrt_llm/_torch/modeling_v2/models/inkling2/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Inkling 2 (`Inkling2ForConditionalGeneration`): ops, kernels and targets of the family."""
203 changes: 203 additions & 0 deletions tensorrt_llm/_torch/modeling_v2/models/inkling2/config.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""The Inkling 2 text decoder's shape, as the family's ops and targets read it.

`InklingDims` is built once from the checkpoint's `config.json` text config and its
`hf_quant_config.json`, and is the only place a target reads model geometry from. It accepts both
`tensorrt_llm._torch.configs.Inkling2TextConfig` and a native Transformers `Inkling2TextConfig`,
which name a few fields differently (`conv_kernel_size` for `sconv_kernel_size`,
`moe_intermediate_size` for `intermediate_size`) and mark local layers by `layer_types` value
`hybrid_sliding` instead of listing `local_layer_ids`.

Expert precision is per layer and comes from `hf_quant_config.json`, whose `parameter_groups` name
every expert tensor with the format it is stored in. The released checkpoint stores layers 0-39
in NVFP4 and layers 40-47 in MXFP8; nothing here assumes that split.
"""

from __future__ import annotations

import enum
import json
import os
import re
from dataclasses import dataclass
from typing import Any, Mapping, Optional

#: The expert weight tensor `hf_quant_config.json` names for each MoE layer.
_EXPERT_W13 = re.compile(r"^model\.llm\.layers\.(\d+)\.mlp\.experts\.w13_weight$")


class ExpertFormat(str, enum.Enum):
"""How a layer's routed experts are stored and how their activations are quantized."""

#: E2M1 values in packed uint8, an FP8 E4M3 scale per 16 values along K, and a per-expert FP32
#: global scale `amax / outer_scale_denominator`. Activations are quantized per token with the
#: 4/6 block-scale rule.
NVFP4 = "nvfp4"
#: FP8 E4M3 values and a UE8M0 scale per 32 values along K; activations are MXFP8 too.
MXFP8 = "mxfp8"
#: BF16 experts, as in a dequantized or randomly initialized test checkpoint.
BF16 = "bfloat16"


def _first(config: Any, *names: str, default: Any = None) -> Any:
for name in names:
value = getattr(config, name, None)
if value is not None:
return value
return default


def expert_formats_from_quant_config(
hf_quant_config: Optional[Mapping[str, Any]], num_layers: int
) -> tuple[ExpertFormat, ...]:
"""Each layer's expert format, from an Inkling `hf_quant_config.json` payload.

A layer that no parameter group names keeps BF16 experts; a layer that two groups name is a
malformed file.
"""
formats: dict[int, ExpertFormat] = {}
groups = ((hf_quant_config or {}).get("model") or {}).get("parameter_groups") or []
for group in groups:
compute = str(group.get("compute_dtype", "")).lower()
fmt = {"nvfp4": ExpertFormat.NVFP4, "mxfp8": ExpertFormat.MXFP8}.get(compute)
if fmt is None:
continue
for tensor in group.get("tensors", []):
match = _EXPERT_W13.match(tensor)
if match is None:
continue
layer = int(match.group(1))
if layer in formats and formats[layer] is not fmt:
raise ValueError(f"hf_quant_config.json gives layer {layer} two expert formats")
formats[layer] = fmt
out_of_range = sorted(i for i in formats if not 0 <= i < num_layers)
if out_of_range:
raise ValueError(
f"hf_quant_config.json names experts of layers {out_of_range}, "
f"outside the {num_layers} decoder layers"
)
Comment thread
moraxu marked this conversation as resolved.
Resolved
return tuple(formats.get(i, ExpertFormat.BF16) for i in range(num_layers))


def read_hf_quant_config(checkpoint_dir: Optional[str]) -> Optional[dict]:
"""The checkpoint's `hf_quant_config.json`, or None when it has none."""
if not checkpoint_dir:
return None
path = os.path.join(checkpoint_dir, "hf_quant_config.json")
if not os.path.isfile(path):
return None
with open(path) as f:
return json.load(f)


@dataclass(frozen=True)
class InklingDims:
"""Geometry of the Inkling 2 text decoder, whole-model (not per rank)."""

hidden_size: int
num_layers: int
vocab_size: int
unpadded_vocab_size: int
num_heads: int
num_kv_heads: int
head_dim: int
d_rel: int
rel_extent: int
sliding_window: int
global_layers: tuple[int, ...]
conv_kernel_size: int
attn_res_block_size: int
num_experts: int
top_k: int
expert_intermediate_size: int
route_scale: float
gate_softcap: float
up_softcap: float
logit_softcap: float
qk_clip_threshold: float
rel_clip_threshold: float
log_scaling_alpha: float
log_scaling_n_floor: float
rms_norm_eps: float
expert_formats: tuple[ExpertFormat, ...]

@classmethod
def from_config(
cls, text_config: Any, hf_quant_config: Optional[Mapping[str, Any]] = None
) -> "InklingDims":
"""Read the text config (either config class) and the quant config payload."""
num_layers = int(text_config.num_hidden_layers)
local = getattr(text_config, "local_layer_ids", None)
if local is not None:
local = set(int(i) for i in local)
global_layers = tuple(i for i in range(num_layers) if i not in local)
else:
layer_types = list(text_config.layer_types)
global_layers = tuple(
i for i, kind in enumerate(layer_types) if "sliding" not in str(kind).lower()
)
swa = (
_first(text_config, "swa_num_attention_heads"),
_first(text_config, "swa_num_key_value_heads"),
_first(text_config, "swa_head_dim"),
)
full = (
text_config.num_attention_heads,
text_config.num_key_value_heads,
text_config.head_dim,
)
if any(s is not None and s != f for s, f in zip(swa, full)):
raise ValueError(
f"sliding-window layers with a different attention shape {swa} from "
f"the global layers {full} are not supported"
)
return cls(
hidden_size=int(text_config.hidden_size),
num_layers=num_layers,
vocab_size=int(text_config.vocab_size),
unpadded_vocab_size=int(
_first(text_config, "unpadded_vocab_size", default=text_config.vocab_size)
),
num_heads=int(text_config.num_attention_heads),
num_kv_heads=int(text_config.num_key_value_heads),
head_dim=int(text_config.head_dim),
d_rel=int(text_config.d_rel),
rel_extent=int(text_config.rel_extent),
sliding_window=int(_first(text_config, "sliding_window_size", "sliding_window")),
global_layers=global_layers,
conv_kernel_size=int(_first(text_config, "sconv_kernel_size", "conv_kernel_size")),
attn_res_block_size=int(text_config.n_layers_per_attn_res_block),
num_experts=int(text_config.n_routed_experts),
top_k=int(text_config.num_experts_per_tok),
expert_intermediate_size=int(
_first(text_config, "moe_intermediate_size", "intermediate_size")
),
route_scale=float(_first(text_config, "route_scale", "routed_scaling_factor")),
gate_softcap=float(text_config.mlp_gate_softcap),
up_softcap=float(text_config.mlp_up_softcap),
logit_softcap=float(text_config.final_logit_softcapping),
qk_clip_threshold=float(text_config.qk_l2_clip_threshold),
rel_clip_threshold=float(text_config.rel_l2_clip_threshold),
log_scaling_alpha=float(text_config.log_scaling_alpha),
log_scaling_n_floor=float(text_config.log_scaling_n_floor),
rms_norm_eps=float(text_config.rms_norm_eps),
expert_formats=expert_formats_from_quant_config(hf_quant_config, num_layers),
)

def is_global(self, layer: int) -> bool:
"""Whether `layer` attends over the whole context rather than the sliding window."""
return layer in self.global_layers

def rel_extent_of(self, layer: int) -> int:
"""Relative distances the layer's position bias covers: the window in a sliding layer."""
return self.rel_extent if self.is_global(layer) else self.sliding_window

def is_block_boundary(self, layer: int) -> bool:
"""Whether `layer` starts a new attention-residual block."""
return layer % self.attn_res_block_size == 0

@property
def num_block_states(self) -> int:
"""Completed blocks the final output mix reads: the embedding plus one per later boundary."""
return len([i for i in range(self.num_layers) if self.is_block_boundary(i)])
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""`KVCacheManagerV2` with Inkling 2's short-convolution windows.

Every decoder layer convolves four streams over the last 4 tokens: its keys and values (this
rank's KV heads) and its attention and MoE outputs (the full hidden width). A request's step
needs the 3 inputs of each stream before its first new token, so the manager keeps them, per
layer and request, in a slot of four tensors `[slots, 3, C]` (bf16): `k`, `v`, `attn`, `mlp`.

A request takes a slot the first time a step names it and keeps it until the engine frees its
KV cache. The slot pool holds `max_batch_size` requests plus the engine's padding and warmup
requests. The windows are not part of the KV cache's prefix reuse, so block reuse must be off.
"""

from __future__ import annotations

from typing import Iterable

import torch

from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import KVCacheManagerV2

from ..config import InklingDims

#: Slots beyond the resident requests: CUDA-graph padding and warmup requests.
SPARE_SLOTS = 8
#: Inputs a width-4 window keeps.
WINDOW = 3


class ConvSlots:
"""Which short-convolution slot each live request owns.

A request keeps its slot, and so its windows, from its first step until it is released, also
while it is paused; a released slot goes to the next new request.
"""

def __init__(self, num_slots: int):
self.num_slots = num_slots
self._slot_of: dict[int, int] = {}
self._free = list(range(num_slots - 1, -1, -1))

def acquire(self, request_ids: Iterable[int]) -> list[int]:
"""Each request's slot; a request seen for the first time gets a free one."""
slots = []
for request_id in request_ids:
slot = self._slot_of.get(request_id)
if slot is None:
if not self._free:
raise RuntimeError(
f"all {self.num_slots} short-convolution slots are in use; "
"more requests are live than the KV cache manager admits"
)
slot = self._free.pop()
self._slot_of[request_id] = slot
slots.append(slot)
return slots

def release(self, request_id: int) -> None:
"""Free the request's slot; a request that never got one is ignored."""
slot = self._slot_of.pop(request_id, None)
if slot is not None:
self._free.append(slot)


class Inkling2KVCacheManager(KVCacheManagerV2):
"""The KV cache plus each request's short-convolution windows, for every rank-local layer.

Block reuse is refused: a reused prefix would come without the convolution windows of the
tokens it covers.
"""

def __init__(
self, *args, pretrained_config=None, mapping=None, max_batch_size: int = 1, **kwargs
):
super().__init__(
*args,
pretrained_config=pretrained_config,
mapping=mapping,
max_batch_size=max_batch_size,
**kwargs,
)
Comment thread
moraxu marked this conversation as resolved.
Resolved
if self.enable_block_reuse:
raise ValueError(
"Inkling2KVCacheManager does not support KV cache block reuse: a reused prefix "
"comes without its short-convolution windows. Set "
"kv_cache_config.enable_block_reuse to false."
)
text = getattr(pretrained_config, "text_config", None) or pretrained_config
dims = InklingDims.from_config(text)
tp = mapping.tp_size if mapping is not None else 1
kv_width = dims.num_kv_heads * dims.head_dim // tp
# Every sequence the manager admits holds its windows: a retiring request keeps its slot
# one more iteration under the overlap scheduler, and a request paused by GEN preemption
# keeps its slot, so the base manager's own count sizes the pool.
self.num_conv_slots = self.max_admissible_sequences + SPARE_SLOTS
widths = {"k": kv_width, "v": kv_width, "attn": dims.hidden_size, "mlp": dims.hidden_size}
device = torch.device("cuda", torch.cuda.current_device())
self._conv_state = [
{
name: torch.zeros(
self.num_conv_slots, WINDOW, width, dtype=torch.bfloat16, device=device
)
for name, width in widths.items()
}
for _ in range(dims.num_layers)
]
self._conv_slots = ConvSlots(self.num_conv_slots)

def conv_state(self, layer: int) -> dict:
"""Layer `layer`'s windows: `k`, `v`, `attn`, `mlp`, each `[slots, 3, C]` bf16."""
return self._conv_state[layer]

def conv_slots(self, request_ids: Iterable[int]) -> torch.Tensor:
"""Each request's slot, as int32 on the device; a request seen for the first time gets one."""
return torch.tensor(self._conv_slots.acquire(request_ids), dtype=torch.int32).to(
self._conv_state[0]["k"].device, non_blocking=True
)

def free_resources(self, request, pin_on_release: bool = False):
self._conv_slots.release(request.py_request_id)
return super().free_resources(request, pin_on_release)
Loading
Loading