Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
c43d0c9
[TRTLLM-17361][test] Add Beam component coverage
Mgluhovskoi Nov 12, 2026
0d1d796
test: group Beam CPU checks with ModelingV2
Mgluhovskoi Nov 12, 2026
a093e31
test: annotate Beam component helpers and test interfaces
Mgluhovskoi Nov 12, 2026
98065d6
fix: accept sufficient Beam sliding-cache retention
Mgluhovskoi Nov 12, 2026
ec73c2f
fix: reject unsupported Beam RoPE configuration
Mgluhovskoi Nov 13, 2026
4586bf2
fix: respect pinned-memory policy in Beam tile reader
Mgluhovskoi Nov 16, 2026
82b4dbc
fix: reuse catalog collectives and checkpoint metadata for Beam
Mgluhovskoi Nov 16, 2026
db9e379
docs: align Beam README with available components and serving features
Mgluhovskoi Nov 16, 2026
95ddc83
fix: allowlist the publicly documented Beam architecture
Mgluhovskoi Nov 16, 2026
e40526b
fix: reject nested Beam checkpoint KV-cache schemes
Mgluhovskoi Nov 16, 2026
d9f5b55
test: verify Beam diagnostic flags and per-instance settings
Mgluhovskoi Nov 16, 2026
5b91a92
test: correct Beam cache readback for nonzero pool slots
Mgluhovskoi Nov 16, 2026
0cdf10a
test: release Beam dense state and isolate reference RNG
Mgluhovskoi Nov 16, 2026
18b26d4
test: check Beam behavior and native router tie order
Mgluhovskoi Nov 16, 2026
deb9458
test: cover Beam paged prefixes beyond sliding window
Mgluhovskoi Nov 16, 2026
98b291e
test: consolidate Beam component helpers and block reference math
Mgluhovskoi Nov 16, 2026
db330e0
test: preserve metadata-only dense operation records
Mgluhovskoi Nov 16, 2026
5f6cdb8
Strengthen Beam component test assertions
Mgluhovskoi Nov 16, 2026
22d1a8c
Report optional Beam source comparisons as separate skipped tests
Mgluhovskoi Nov 16, 2026
ba108e0
Move Beam component evidence reporting to opt-in QA runner
Mgluhovskoi Nov 16, 2026
dbc8725
Correct new-token context length in cached-prefix test fixture
Mgluhovskoi Nov 16, 2026
3b4f2dc
[refactor] Remove unused Beam component interfaces
Mgluhovskoi Nov 16, 2026
05c3b83
[docs] Clarify Beam family operator references
Mgluhovskoi Nov 16, 2026
7295d5e
[fix] Validate Beam routing IDs and block cache retention
Mgluhovskoi Nov 17, 2026
86937ed
Align Beam TP1 block tests with served normalization and activation
Mgluhovskoi Nov 17, 2026
ba79122
Account for router projection in Beam sparse operator trace
Mgluhovskoi Nov 17, 2026
001c66f
Document shared serving operators in Beam test helpers
Mgluhovskoi Nov 17, 2026
82ed6c9
[infra] Complete Beam ModelingV2 package migration
Mgluhovskoi Nov 17, 2026
44fb476
Skip Beam component suites on unsupported CUDA devices
Mgluhovskoi Nov 18, 2026
479e66d
Isolate collectives in single-GPU Beam weight-state test
Mgluhovskoi Nov 18, 2026
a98df36
Move manual Beam qualification suites into integration tests
Mgluhovskoi Nov 18, 2026
4a9b060
test: simplify redundant Beam component coverage
Mgluhovskoi Nov 18, 2026
43cf5fc
docs: keep B300 test timing comments platform-specific
Mgluhovskoi Nov 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/source/models/supported-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ following its [Docker guide](https://github.com/NVIDIA/TensorRT-LLM/blob/main/do
| ----- | ----------- | ---------------- | ----------------- |
| MiniMax-M3.1-Flash | [`feat/minimax-m3.1-flash`](https://github.com/NVIDIA/TensorRT-LLM/tree/feat/minimax-m3.1-flash) | [Serving MiniMax-M3.1-Flash](https://github.com/NVIDIA/TensorRT-LLM/blob/feat/minimax-m3.1-flash/docs/source/deployment-guide/deployment-guide-for-minimax-m3-on-trtllm.md#serving-minimax-m31-flash) in the MiniMax-M3 deployment guide | `nvcr.io/nvidia/tensorrt-llm/release:feat-minimax-m3.1-flash-1.4.0a0.dev202611050719` |
| Kimi K3.1 Pro | [`feat/kimi-k3.1-pro`](https://github.com/NVIDIA/TensorRT-LLM/tree/feat/kimi-k3.1-pro) | [Kimi K3.1 Pro deployment guide](https://github.com/NVIDIA/TensorRT-LLM/blob/feat/kimi-k3.1-pro/docs/source/deployment-guide/deployment-guide-for-kimi-k3.1-pro-on-trtllm.md) | `nvcr.io/nvidia/tensorrt-llm/release:feat-kimi-k3.1-pro-1.4.0a0.dev202611050000` |
| Beam (Reflection AI) | [`feat/reflection-beam`](https://github.com/NVIDIA/TensorRT-LLM/tree/feat/reflection-beam) | [Beam deployment guide](https://github.com/NVIDIA/TensorRT-LLM/blob/feat/reflection-beam/docs/source/deployment-guide/deployment-guide-for-beam-on-trtllm.md) | No container published yet — build from the branch sources per the guide |
| Beam (Reflection AI), `BeamForCausalLM` ([public NVFP4 checkpoint](https://huggingface.co/reflection/Beam-1.0-501B-A23B-NVFP4)) | [`feat/reflection-beam`](https://github.com/NVIDIA/TensorRT-LLM/tree/feat/reflection-beam) | [Beam deployment guide](https://github.com/NVIDIA/TensorRT-LLM/blob/feat/reflection-beam/docs/source/deployment-guide/deployment-guide-for-beam-on-trtllm.md) | No container published yet — build from the branch sources per the guide |

## Model-Feature Support Matrix (Key Models)

Expand Down
88 changes: 63 additions & 25 deletions tensorrt_llm/_torch/modeling_v2/models/beam/README.md
Original file line number Diff line number Diff line change
@@ -1,36 +1,74 @@
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: Apache-2.0 -->

# Beam StairCase components
# Beam components with ModelingV2

This directory implements Beam components using the experimental ModelingV2/StairCase
architecture: family-scoped operators and a 52-layer NVFP4-expert component for attention TP4 and MoE TP4/EP1 on four GB300 GPUs.
This directory contains Beam family operators and a directly constructed 52-layer
NVFP4-expert component. It does not register an engine target: `LLM()` and
`trtllm-serve` cannot select these components through ModelingV2. Component users
construct `BeamFullModelNvfp4Sm103Tp4` from `full_model_nvfp4__sm_103__tp4/model.py`
and provide the checkpoint configuration, rank mapping and device; the component's
`load()` method loads the caller-provided checkpoint before execution. The tests
below exercise the component APIs and their input contracts.

The full component implements sliding-window and full attention with a BF16 KV cache.
Its loader streams caller-provided weights into rank-local storage; no checkpoint is
included. Router projection and selection use FP32, and packed NVFP4 experts use the
native GEMM path.
## Configuration and limits

These are directly constructed components, not a registered `LLM()` or `trtllm-serve`
target. Engine routing, the engine adapter, checkpoint-loader registration and a named
whole-model accuracy gate belong to the serving integration. Operator certification and
CPU component-contract tests accompany these components; broader component tests are
reviewed separately. Speculative decoding, FP8 KV cache, disaggregated serving, CUDA
graphs and overlap scheduling are outside this component baseline.
| Setting | Component configuration |
|---|---|
| Hardware and layout | Four GB300 GPUs (SM103), attention TP4, MoE TP4/EP1, PP1 |
| Parameters | Packed NVFP4 experts; BF16 dense tensors; FP32 router projection and selection |
| Attention | Checkpoint-defined sliding-window and full attention, BF16 KV cache |
| Execution | Eager; CUDA graphs and overlap scheduling disabled |

Beam-only operators live under `ops/`, with tests in the matching Beam test tree.
Component-specific cases for shared catalog operators reuse the catalog test drivers,
including TP1 shapes and the per-rank TP4 projection, attention and expert shapes.
Checkpoint weights are supplied separately and streamed into rank-local storage.
No checkpoint, reference captures or private evaluation artifacts are included.
Other GPU architectures and parallel layouts, speculative decoding, FP8 KV cache,
disaggregated serving, CUDA graphs and overlap scheduling are outside this baseline.

## Checkout-only bring-up helpers
## File organization

The six TP1 bring-up components and their generation controller live with their tests in
`tests/unittest/_torch/modeling_v2/models/beam/_components/`. They are not installed in
the TensorRT-LLM wheel. The runtime does not expose intermediate-tensor capture hooks.
Normal runtime validation and rank-consistency diagnostics remain available.
- `ops/` contains Beam-only operators. Their tests mirror that directory under
`tests/unittest/_torch/modeling_v2/models/beam/ops/`.
- `_tp4/` contains shared BF16 layout, loading and component base modules.
- `full_model_nvfp4__sm_103__tp4/` adds packed expert storage and full-depth execution.
- `tests/unittest/_torch/modeling_v2/models/beam/_components/` contains checkout-only
TP1 bring-up helpers, including the six-layer model and its generation controller.
They are not installed in the wheel. Intermediate capture instrumentation lives
in tests; runtime validation and rank-consistency diagnostics remain in the library.

The TP4 checkpoint layout, expert provider and full-model execution remain in this
library. Public serving integration uses `LLM()` and `trtllm-serve`.
## Run component tests

Shared BF16 TP4 layout, loading and component base modules live in `_tp4/`; the
`full_model_nvfp4__sm_103__tp4/` package adds NVFP4 expert storage and execution.
Run from the repository root with a matching TensorRT-LLM installation and its
native dependencies. `cpu_only` selects tests that do not execute GPU operations;
it does not make the TensorRT-LLM package independent of its native dependencies.

```bash
python -m pytest -m cpu_only \
tests/unittest/_torch/modeling_v2/models/beam/test_component_contracts.py \
tests/unittest/_torch/modeling_v2/models/beam/test_diagnostic_mode.py \
tests/unittest/_torch/modeling_v2/models/beam/full_model_nvfp4__sm_103__tp4/test_checkpoint_layout.py
```

For generated-input operator coverage on one compatible GPU:

```bash
python -m pytest tests/unittest/_torch/modeling_v2/models/beam/ops
```

For the small four-rank load-error and collective checks, with four GPUs visible
and working MPI peer communication:

```bash
env -u TLLM_DISABLE_MPI mpirun -n 4 python -m pytest -q \
tests/unittest/_torch/modeling_v2/models/beam/test_distributed_loading.py
```

These checks use generated data, not a checkpoint. The collective test includes
capture/replay of the communication operation; this does not qualify whole-model
CUDA graphs. Component checks do not establish whole-model accuracy.

CPU and bounded single-GPU CI coverage is selected explicitly in
the test lists under `tests/integration/test_lists/test-db/`, including
`l0_b300.yml` for GPU operators.
Large component and four-rank checks are manual; do not run the entire Beam test
directory as a lightweight smoke test. Full GSM8K is not part of CI.
20 changes: 19 additions & 1 deletion tensorrt_llm/_torch/modeling_v2/models/beam/_attention.py
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,7 @@ def check_step(
head_dim: int,
device: torch.device,
max_tokens: int,
attention_window: int | None,
) -> None:
"""Raise ValueError on any call input a block would otherwise compute wrongly.

Expand All @@ -183,7 +184,8 @@ def check_step(
``arange(past, past + T)`` on that device; a prepared ``TrtllmAttentionMetadata``
holding one step of one request on a BF16 ``KVCacheManagerV2`` that holds logical
layer ``layer_idx`` with ``num_kv_heads`` kv heads of ``head_dim`` (``check_metadata``
states the steps and returns ``past``).
states the steps and returns ``past``). Cache retention must cover ``attention_window``
(or the full sequence for None), capped by the manager's maximum sequence length.
"""
if not isinstance(hidden_states, torch.Tensor) or hidden_states.dtype != torch.bfloat16:
raise ValueError("hidden_states must be a torch.bfloat16 tensor")
Expand All @@ -204,6 +206,22 @@ def check_step(
past = check_metadata(
md, num_tokens, layer_idx=layer_idx, num_kv_heads=num_kv_heads, head_dim=head_dim
)
manager = md.kv_cache_manager
slot = manager.layer_offsets[layer_idx]
windows = manager.max_attention_window_vec
if slot >= len(windows):
raise ValueError(f"the KV cache manager has no retention window for layer {layer_idx}")
retained = windows[slot]
required = (
manager.max_seq_len
if attention_window is None
else min(attention_window, manager.max_seq_len)
)
if retained is not None and int(retained) < required:
raise ValueError(
f"layer {layer_idx} requires cache retention of at least {required} tokens; "
f"the cache keeps a window of {retained}"
)
expected = torch.arange(
past, past + num_tokens, dtype=position_ids.dtype, device=position_ids.device
)
Expand Down
37 changes: 28 additions & 9 deletions tensorrt_llm/_torch/modeling_v2/models/beam/_tp4/component_bf16.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

Per call, rank r computes its 12 query heads / 2 kv heads, its dense or expert intermediate
slice and its vocabulary rows; the enclosing method applies exactly one collective at each
boundary, on the TRT-LLM NCCL all-reduce or all-gather::
boundary, through the ModelingV2 catalog all-reduce or all-gather::

embed masked local lookup, one all-reduce (sum)
attention local q/k/v/gate, q/k norms, RoPE or NoPE scale, TRTLLM attention on the
Expand All @@ -37,17 +37,19 @@
import json
import os
from collections import OrderedDict
from functools import partial
from typing import Any, Mapping, Optional, Sequence

import torch
from torch import nn

from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttentionMetadata
from tensorrt_llm._torch.distributed import AllReduce, allgather
from tensorrt_llm._torch.modeling_v2._router_index import (
MODELING_V2_VALIDATE_ENV,
step_contract_enabled,
)
from tensorrt_llm._torch.modeling_v2.catalog.comm.allgather import allgather
from tensorrt_llm._torch.modeling_v2.catalog.comm.allreduce import allreduce, allreduce_group_args
from tensorrt_llm._torch.modeling_v2.catalog.gemm.cublas_mm import cublas_mm
from tensorrt_llm._torch.modeling_v2.catalog.norm.flashinfer_rmsnorm import flashinfer_rmsnorm
from tensorrt_llm._torch.modeling_v2.models.beam.ops.activation.beam_logit_softcap import (
Expand All @@ -62,7 +64,7 @@
beam_depth_residual,
depth_scales,
)
from tensorrt_llm.functional import AllReduceStrategy
from tensorrt_llm.functional import AllReduceFusionOp
from tensorrt_llm.mapping import Mapping as ParallelMapping

from .. import _attention, dispatch_combine
Expand Down Expand Up @@ -161,11 +163,28 @@ def __init__(
self.softcap = config["final_logit_softcapping"]
self.vocab_interval = self.specs[EMBED].interval(self.rank)[1:]
self._allocate_storage()
self._allreduce = AllReduce(mapping=mapping, strategy=AllReduceStrategy.NCCL)
self._init_collectives()
self.usable = False
self.load_record: Optional[dict] = None
self._init_diagnostics()

def _init_collectives(self) -> None:
args = allreduce_group_args(self.mapping)
self._allreduce = partial(
allreduce,
residual=None,
norm_weight=None,
scale=None,
bias=None,
op=int(AllReduceFusionOp.NONE),
eps=0.0,
trigger_completion_at_end=False,
**args,
)
self._allgather = partial(
allgather, sizes=None, group=args["group"], process_group=args["process_group"]
)

def _allocate_storage(self) -> None:
#: Storage is always resolved from the currently registered parameters, never from
#: references taken here, so a replaced parameter or bank is what every call uses.
Expand Down Expand Up @@ -206,8 +225,8 @@ def _init_diagnostics(self) -> None:

def _check_diagnostics_agree(self) -> None:
"""Raise on every rank unless all four ranks run the same diagnostic mode (collective)."""
flag = torch.tensor([1.0 if self.diagnostics else 0.0], device=self.device)
on = int(self._allreduce(flag).item())
flag = torch.tensor([int(self.diagnostics)], dtype=torch.int32, device=self.device)
on = int(self._allgather(flag).sum().item())
if on not in (0, WORLD_SIZE):
raise LoadError(
f"rank {self.rank}: diagnostic validation is on for {on} of {WORLD_SIZE} ranks; "
Expand Down Expand Up @@ -361,7 +380,7 @@ def _bank(self, layer: int, which: str) -> torch.Tensor:
return self.bank(f"layers.{layer}.experts.{which}")

def all_reduce(self, tensor: torch.Tensor) -> torch.Tensor:
"""The one sum over the four ranks at a partial-result boundary (NCCL)."""
"""The one sum over the four ranks at a partial-result boundary (catalog AUTO)."""
return self._allreduce(tensor)

def _digests_agree(self, tensors: Sequence[torch.Tensor]) -> list:
Expand All @@ -376,7 +395,7 @@ def _digests_agree(self, tensors: Sequence[torch.Tensor]) -> list:
h.update(memoryview(t.contiguous().cpu().view(torch.uint8).numpy().reshape(-1)))
digests.append(h.digest())
local = torch.frombuffer(bytearray(b"".join(digests)), dtype=torch.uint8).to(self.device)
gathered = allgather(local, self.mapping, dim=0).view(WORLD_SIZE, -1).cpu()
gathered = self._allgather(local).view(WORLD_SIZE, -1).cpu()
if not all(torch.equal(gathered[0], gathered[r]) for r in range(1, WORLD_SIZE)):
raise RuntimeError(
"replicated MoE routing differs between ranks (input, ids or weights)"
Expand Down Expand Up @@ -562,7 +581,7 @@ def head(self, n: torch.Tensor) -> torch.Tensor:
self._check_usable()
_check_hidden(n, self.hidden, self.device, self.max_tokens)
local = cublas_mm(n, self.tensor(HEAD).t())
gathered = allgather(local, self.mapping, dim=-1)
gathered = self._allgather(local.t().contiguous()).t().contiguous()
logits = beam_logit_softcap(gathered, self.softcap)
return logits

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,8 @@ def check_config(config: Any) -> Schedule:
and mlp.index("sparse") < len(mlp) - mlp[::-1].index("dense")
):
raise ValueError("dense feed-forward layers must precede the sparse ones")
if _get(config, "rope_scaling") is not None:
raise ValueError("rope_scaling is unsupported; sliding layers use unscaled RoPE")
rope = _get(config, "rope_parameters")
sliding = (rope or {}).get("sliding_attention") if isinstance(rope, Mapping) else None
if (
Expand Down
Loading
Loading