Repository navigation
[None][feat] Add the Inkling 2 Modeling V2 family: operators, kernels and the Inkling-2-Small target #22627
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
moraxu
merged 4 commits into
NVIDIA:main
from
moraxu:dev-mguzek-inkling2-modeling-v2-family
Nov 20, 2026
+5,077
−1
Merged
[None][feat] Add the Inkling 2 Modeling V2 family: operators, kernels and the Inkling-2-Small target #22627
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
9b4c640
[None][feat] Add the Inkling 2 Modeling V2 family: operators, kernels…
moraxu e2baee7
[None][fix] Inkling 2 family: input guards, conv slots and tests
moraxu 8f9ff78
[None][test] Test the convolution slots with GEN preemption headroom
moraxu f45d2c2
[None][fix] Inkling 2: slot pool from the manager's count; check page…
moraxu File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| <!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> | ||
| <!-- SPDX-License-Identifier: Apache-2.0 --> | ||
|
|
||
| # Inkling 2 | ||
|
|
||
| `Inkling2ForConditionalGeneration` (`thinkingmachines/Inkling-2-Small`): a 48-layer MoE text decoder | ||
| (hidden 4608, 384 routed experts, top 8) with a vision tower. The family implements the text model. | ||
|
|
||
| ## What the architecture needs | ||
|
|
||
| * **Attention without rotary embeddings.** Position enters through a relative bias each token mixes | ||
| per head from 16 bias-vs-distance profiles (`rel_logits_proj`), over distances below 1024. 40 | ||
| sliding-window layers attend the last 1024 tokens; 8 global layers (5, 11, ..., 47) attend the | ||
| whole prefix and scale q and the relative states by `tau = 1 + 0.1 ln(max((p + 1) / 8192, 1))`. | ||
| Each token and head has a sink logit, and the output is gated by `sigmoid(gate)`. q, k and the | ||
| relative states are L2-clipped; v is RMS-normalized without a weight. | ||
| * **Short convolutions.** Width-4 causal convolutions with a residual run on k and v (dynamic | ||
| per-token taps on the global layers) and on each sublayer's output. Their windows are | ||
| per-request state. | ||
| * **Block attention residuals.** Every 6 layers the residual stream closes a block; each | ||
| sublayer reads a learned softmax mix of the normalized blocks and the current partial block. | ||
| * **Experts.** fp32 sigmoid router with a selection bias; SiTU activation | ||
| (`4 tanh(g / 4) sigmoid(g) * 25 tanh(u / 25)`). The checkpoint stores layers 0-39 in NVFP4 (with | ||
| per-token 4/6 NVFP4 activations) and layers 40-47 in MXFP8. | ||
|
|
||
| ## Layout | ||
|
|
||
| ```text | ||
| inkling2/ | ||
| ├── config.py # InklingDims: the decoder's shape and per-layer expert formats | ||
| ├── kernels/ # Triton: short convolution, paged attention with the relative bias | ||
| ├── ops/ # OpWrappers: conv, attention prologue and core, residual mix, | ||
| │ # router, NVFP4 experts (FlashInfer CuTe DSL), MXFP8 experts | ||
| └── inkling_2_small_nvfp4__sm_103__tep4/ | ||
| ├── weights.py # weight table: TP4 heads, EP4 experts, kernel layouts | ||
| ├── modeling.py # the 48-layer forward and the engine shell | ||
| └── cache.py # KVCacheManagerV2 with the convolution windows | ||
| ``` | ||
|
|
||
| ## Target `inkling_2_small_nvfp4__sm_103__tep4` | ||
|
|
||
| The released checkpoint on one GB300 tray: attention TP4 (12 query and 2 KV heads per rank), MoE | ||
| EP4 (96 experts per rank), BF16 KV cache. The baseline runs eagerly, without CUDA graphs, the | ||
| overlap scheduler, chunked prefill, KV cache block reuse or speculative decoding. | ||
|
|
||
| The family is not routed yet: these are the operators and the target's components; the route that | ||
| serves the target through `LLM()` and `trtllm-serve` follows separately. | ||
|
|
||
| The NVFP4 experts run FlashInfer's CuTe DSL fused MoE with SiTU and per-token 4/6 activations. The | ||
| MXFP8 experts run a grouped Triton MoE that expands the E4M3 weights to bf16 in registers (exactly) | ||
| and accumulates in fp32. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| """Inkling 2 (`Inkling2ForConditionalGeneration`): ops, kernels and targets of the family.""" |
203 changes: 203 additions & 0 deletions
203
tensorrt_llm/_torch/modeling_v2/models/inkling2/config.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,203 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| """The Inkling 2 text decoder's shape, as the family's ops and targets read it. | ||
|
|
||
| `InklingDims` is built once from the checkpoint's `config.json` text config and its | ||
| `hf_quant_config.json`, and is the only place a target reads model geometry from. It accepts both | ||
| `tensorrt_llm._torch.configs.Inkling2TextConfig` and a native Transformers `Inkling2TextConfig`, | ||
| which name a few fields differently (`conv_kernel_size` for `sconv_kernel_size`, | ||
| `moe_intermediate_size` for `intermediate_size`) and mark local layers by `layer_types` value | ||
| `hybrid_sliding` instead of listing `local_layer_ids`. | ||
|
|
||
| Expert precision is per layer and comes from `hf_quant_config.json`, whose `parameter_groups` name | ||
| every expert tensor with the format it is stored in. The released checkpoint stores layers 0-39 | ||
| in NVFP4 and layers 40-47 in MXFP8; nothing here assumes that split. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import enum | ||
| import json | ||
| import os | ||
| import re | ||
| from dataclasses import dataclass | ||
| from typing import Any, Mapping, Optional | ||
|
|
||
| #: The expert weight tensor `hf_quant_config.json` names for each MoE layer. | ||
| _EXPERT_W13 = re.compile(r"^model\.llm\.layers\.(\d+)\.mlp\.experts\.w13_weight$") | ||
|
|
||
|
|
||
| class ExpertFormat(str, enum.Enum): | ||
| """How a layer's routed experts are stored and how their activations are quantized.""" | ||
|
|
||
| #: E2M1 values in packed uint8, an FP8 E4M3 scale per 16 values along K, and a per-expert FP32 | ||
| #: global scale `amax / outer_scale_denominator`. Activations are quantized per token with the | ||
| #: 4/6 block-scale rule. | ||
| NVFP4 = "nvfp4" | ||
| #: FP8 E4M3 values and a UE8M0 scale per 32 values along K; activations are MXFP8 too. | ||
| MXFP8 = "mxfp8" | ||
| #: BF16 experts, as in a dequantized or randomly initialized test checkpoint. | ||
| BF16 = "bfloat16" | ||
|
|
||
|
|
||
| def _first(config: Any, *names: str, default: Any = None) -> Any: | ||
| for name in names: | ||
| value = getattr(config, name, None) | ||
| if value is not None: | ||
| return value | ||
| return default | ||
|
|
||
|
|
||
| def expert_formats_from_quant_config( | ||
| hf_quant_config: Optional[Mapping[str, Any]], num_layers: int | ||
| ) -> tuple[ExpertFormat, ...]: | ||
| """Each layer's expert format, from an Inkling `hf_quant_config.json` payload. | ||
|
|
||
| A layer that no parameter group names keeps BF16 experts; a layer that two groups name is a | ||
| malformed file. | ||
| """ | ||
| formats: dict[int, ExpertFormat] = {} | ||
| groups = ((hf_quant_config or {}).get("model") or {}).get("parameter_groups") or [] | ||
| for group in groups: | ||
| compute = str(group.get("compute_dtype", "")).lower() | ||
| fmt = {"nvfp4": ExpertFormat.NVFP4, "mxfp8": ExpertFormat.MXFP8}.get(compute) | ||
| if fmt is None: | ||
| continue | ||
| for tensor in group.get("tensors", []): | ||
| match = _EXPERT_W13.match(tensor) | ||
| if match is None: | ||
| continue | ||
| layer = int(match.group(1)) | ||
| if layer in formats and formats[layer] is not fmt: | ||
| raise ValueError(f"hf_quant_config.json gives layer {layer} two expert formats") | ||
| formats[layer] = fmt | ||
| out_of_range = sorted(i for i in formats if not 0 <= i < num_layers) | ||
| if out_of_range: | ||
| raise ValueError( | ||
| f"hf_quant_config.json names experts of layers {out_of_range}, " | ||
| f"outside the {num_layers} decoder layers" | ||
| ) | ||
| return tuple(formats.get(i, ExpertFormat.BF16) for i in range(num_layers)) | ||
|
|
||
|
|
||
| def read_hf_quant_config(checkpoint_dir: Optional[str]) -> Optional[dict]: | ||
| """The checkpoint's `hf_quant_config.json`, or None when it has none.""" | ||
| if not checkpoint_dir: | ||
| return None | ||
| path = os.path.join(checkpoint_dir, "hf_quant_config.json") | ||
| if not os.path.isfile(path): | ||
| return None | ||
| with open(path) as f: | ||
| return json.load(f) | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class InklingDims: | ||
| """Geometry of the Inkling 2 text decoder, whole-model (not per rank).""" | ||
|
|
||
| hidden_size: int | ||
| num_layers: int | ||
| vocab_size: int | ||
| unpadded_vocab_size: int | ||
| num_heads: int | ||
| num_kv_heads: int | ||
| head_dim: int | ||
| d_rel: int | ||
| rel_extent: int | ||
| sliding_window: int | ||
| global_layers: tuple[int, ...] | ||
| conv_kernel_size: int | ||
| attn_res_block_size: int | ||
| num_experts: int | ||
| top_k: int | ||
| expert_intermediate_size: int | ||
| route_scale: float | ||
| gate_softcap: float | ||
| up_softcap: float | ||
| logit_softcap: float | ||
| qk_clip_threshold: float | ||
| rel_clip_threshold: float | ||
| log_scaling_alpha: float | ||
| log_scaling_n_floor: float | ||
| rms_norm_eps: float | ||
| expert_formats: tuple[ExpertFormat, ...] | ||
|
|
||
| @classmethod | ||
| def from_config( | ||
| cls, text_config: Any, hf_quant_config: Optional[Mapping[str, Any]] = None | ||
| ) -> "InklingDims": | ||
| """Read the text config (either config class) and the quant config payload.""" | ||
| num_layers = int(text_config.num_hidden_layers) | ||
| local = getattr(text_config, "local_layer_ids", None) | ||
| if local is not None: | ||
| local = set(int(i) for i in local) | ||
| global_layers = tuple(i for i in range(num_layers) if i not in local) | ||
| else: | ||
| layer_types = list(text_config.layer_types) | ||
| global_layers = tuple( | ||
| i for i, kind in enumerate(layer_types) if "sliding" not in str(kind).lower() | ||
| ) | ||
| swa = ( | ||
| _first(text_config, "swa_num_attention_heads"), | ||
| _first(text_config, "swa_num_key_value_heads"), | ||
| _first(text_config, "swa_head_dim"), | ||
| ) | ||
| full = ( | ||
| text_config.num_attention_heads, | ||
| text_config.num_key_value_heads, | ||
| text_config.head_dim, | ||
| ) | ||
| if any(s is not None and s != f for s, f in zip(swa, full)): | ||
| raise ValueError( | ||
| f"sliding-window layers with a different attention shape {swa} from " | ||
| f"the global layers {full} are not supported" | ||
| ) | ||
| return cls( | ||
| hidden_size=int(text_config.hidden_size), | ||
| num_layers=num_layers, | ||
| vocab_size=int(text_config.vocab_size), | ||
| unpadded_vocab_size=int( | ||
| _first(text_config, "unpadded_vocab_size", default=text_config.vocab_size) | ||
| ), | ||
| num_heads=int(text_config.num_attention_heads), | ||
| num_kv_heads=int(text_config.num_key_value_heads), | ||
| head_dim=int(text_config.head_dim), | ||
| d_rel=int(text_config.d_rel), | ||
| rel_extent=int(text_config.rel_extent), | ||
| sliding_window=int(_first(text_config, "sliding_window_size", "sliding_window")), | ||
| global_layers=global_layers, | ||
| conv_kernel_size=int(_first(text_config, "sconv_kernel_size", "conv_kernel_size")), | ||
| attn_res_block_size=int(text_config.n_layers_per_attn_res_block), | ||
| num_experts=int(text_config.n_routed_experts), | ||
| top_k=int(text_config.num_experts_per_tok), | ||
| expert_intermediate_size=int( | ||
| _first(text_config, "moe_intermediate_size", "intermediate_size") | ||
| ), | ||
| route_scale=float(_first(text_config, "route_scale", "routed_scaling_factor")), | ||
| gate_softcap=float(text_config.mlp_gate_softcap), | ||
| up_softcap=float(text_config.mlp_up_softcap), | ||
| logit_softcap=float(text_config.final_logit_softcapping), | ||
| qk_clip_threshold=float(text_config.qk_l2_clip_threshold), | ||
| rel_clip_threshold=float(text_config.rel_l2_clip_threshold), | ||
| log_scaling_alpha=float(text_config.log_scaling_alpha), | ||
| log_scaling_n_floor=float(text_config.log_scaling_n_floor), | ||
| rms_norm_eps=float(text_config.rms_norm_eps), | ||
| expert_formats=expert_formats_from_quant_config(hf_quant_config, num_layers), | ||
| ) | ||
|
|
||
| def is_global(self, layer: int) -> bool: | ||
| """Whether `layer` attends over the whole context rather than the sliding window.""" | ||
| return layer in self.global_layers | ||
|
|
||
| def rel_extent_of(self, layer: int) -> int: | ||
| """Relative distances the layer's position bias covers: the window in a sliding layer.""" | ||
| return self.rel_extent if self.is_global(layer) else self.sliding_window | ||
|
|
||
| def is_block_boundary(self, layer: int) -> bool: | ||
| """Whether `layer` starts a new attention-residual block.""" | ||
| return layer % self.attn_res_block_size == 0 | ||
|
|
||
| @property | ||
| def num_block_states(self) -> int: | ||
| """Completed blocks the final output mix reads: the embedding plus one per later boundary.""" | ||
| return len([i for i in range(self.num_layers) if self.is_block_boundary(i)]) | ||
2 changes: 2 additions & 0 deletions
2
...rt_llm/_torch/modeling_v2/models/inkling2/inkling_2_small_nvfp4__sm_103__tep4/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 |
122 changes: 122 additions & 0 deletions
122
tensorrt_llm/_torch/modeling_v2/models/inkling2/inkling_2_small_nvfp4__sm_103__tep4/cache.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| """`KVCacheManagerV2` with Inkling 2's short-convolution windows. | ||
|
|
||
| Every decoder layer convolves four streams over the last 4 tokens: its keys and values (this | ||
| rank's KV heads) and its attention and MoE outputs (the full hidden width). A request's step | ||
| needs the 3 inputs of each stream before its first new token, so the manager keeps them, per | ||
| layer and request, in a slot of four tensors `[slots, 3, C]` (bf16): `k`, `v`, `attn`, `mlp`. | ||
|
|
||
| A request takes a slot the first time a step names it and keeps it until the engine frees its | ||
| KV cache. The slot pool holds `max_batch_size` requests plus the engine's padding and warmup | ||
| requests. The windows are not part of the KV cache's prefix reuse, so block reuse must be off. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from typing import Iterable | ||
|
|
||
| import torch | ||
|
|
||
| from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import KVCacheManagerV2 | ||
|
|
||
| from ..config import InklingDims | ||
|
|
||
| #: Slots beyond the resident requests: CUDA-graph padding and warmup requests. | ||
| SPARE_SLOTS = 8 | ||
| #: Inputs a width-4 window keeps. | ||
| WINDOW = 3 | ||
|
|
||
|
|
||
| class ConvSlots: | ||
| """Which short-convolution slot each live request owns. | ||
|
|
||
| A request keeps its slot, and so its windows, from its first step until it is released, also | ||
| while it is paused; a released slot goes to the next new request. | ||
| """ | ||
|
|
||
| def __init__(self, num_slots: int): | ||
| self.num_slots = num_slots | ||
| self._slot_of: dict[int, int] = {} | ||
| self._free = list(range(num_slots - 1, -1, -1)) | ||
|
|
||
| def acquire(self, request_ids: Iterable[int]) -> list[int]: | ||
| """Each request's slot; a request seen for the first time gets a free one.""" | ||
| slots = [] | ||
| for request_id in request_ids: | ||
| slot = self._slot_of.get(request_id) | ||
| if slot is None: | ||
| if not self._free: | ||
| raise RuntimeError( | ||
| f"all {self.num_slots} short-convolution slots are in use; " | ||
| "more requests are live than the KV cache manager admits" | ||
| ) | ||
| slot = self._free.pop() | ||
| self._slot_of[request_id] = slot | ||
| slots.append(slot) | ||
| return slots | ||
|
|
||
| def release(self, request_id: int) -> None: | ||
| """Free the request's slot; a request that never got one is ignored.""" | ||
| slot = self._slot_of.pop(request_id, None) | ||
| if slot is not None: | ||
| self._free.append(slot) | ||
|
|
||
|
|
||
| class Inkling2KVCacheManager(KVCacheManagerV2): | ||
| """The KV cache plus each request's short-convolution windows, for every rank-local layer. | ||
|
|
||
| Block reuse is refused: a reused prefix would come without the convolution windows of the | ||
| tokens it covers. | ||
| """ | ||
|
|
||
| def __init__( | ||
| self, *args, pretrained_config=None, mapping=None, max_batch_size: int = 1, **kwargs | ||
| ): | ||
| super().__init__( | ||
| *args, | ||
| pretrained_config=pretrained_config, | ||
| mapping=mapping, | ||
| max_batch_size=max_batch_size, | ||
| **kwargs, | ||
| ) | ||
|
moraxu marked this conversation as resolved.
Resolved
|
||
| if self.enable_block_reuse: | ||
| raise ValueError( | ||
| "Inkling2KVCacheManager does not support KV cache block reuse: a reused prefix " | ||
| "comes without its short-convolution windows. Set " | ||
| "kv_cache_config.enable_block_reuse to false." | ||
| ) | ||
| text = getattr(pretrained_config, "text_config", None) or pretrained_config | ||
| dims = InklingDims.from_config(text) | ||
| tp = mapping.tp_size if mapping is not None else 1 | ||
| kv_width = dims.num_kv_heads * dims.head_dim // tp | ||
| # Every sequence the manager admits holds its windows: a retiring request keeps its slot | ||
| # one more iteration under the overlap scheduler, and a request paused by GEN preemption | ||
| # keeps its slot, so the base manager's own count sizes the pool. | ||
| self.num_conv_slots = self.max_admissible_sequences + SPARE_SLOTS | ||
| widths = {"k": kv_width, "v": kv_width, "attn": dims.hidden_size, "mlp": dims.hidden_size} | ||
| device = torch.device("cuda", torch.cuda.current_device()) | ||
| self._conv_state = [ | ||
| { | ||
| name: torch.zeros( | ||
| self.num_conv_slots, WINDOW, width, dtype=torch.bfloat16, device=device | ||
| ) | ||
| for name, width in widths.items() | ||
| } | ||
| for _ in range(dims.num_layers) | ||
| ] | ||
| self._conv_slots = ConvSlots(self.num_conv_slots) | ||
|
|
||
| def conv_state(self, layer: int) -> dict: | ||
| """Layer `layer`'s windows: `k`, `v`, `attn`, `mlp`, each `[slots, 3, C]` bf16.""" | ||
| return self._conv_state[layer] | ||
|
|
||
| def conv_slots(self, request_ids: Iterable[int]) -> torch.Tensor: | ||
| """Each request's slot, as int32 on the device; a request seen for the first time gets one.""" | ||
| return torch.tensor(self._conv_slots.acquire(request_ids), dtype=torch.int32).to( | ||
| self._conv_state[0]["k"].device, non_blocking=True | ||
| ) | ||
|
|
||
| def free_resources(self, request, pin_on_release: bool = False): | ||
| self._conv_slots.release(request.py_request_id) | ||
| return super().free_resources(request, pin_on_release) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.