Skip to content

[None][feat] Qwen3.8-Flash-Next FP8 modeling_v2: one target for TP1/TP2/TEP2/DEP2/TP4/TEP4/DEP4 on GB300 - #21621

Merged
Wanli-Jiang merged 51 commits into
NVIDIA:mainfrom
Wanli-Jiang:user/williamj/qwen38fn-fp8-gb300-mv2
Nov 17, 2026
Merged

Wanli-Jiang merged 51 commits into
NVIDIA:mainfrom
Wanli-Jiang:user/williamj/qwen38fn-fp8-gb300-mv2

Conversation

@Wanli-Jiang

@Wanli-Jiang Wanli-Jiang commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Qwen3.8-Flash-Next-FP8 (Qwen4ExpForConditionalGeneration; routed experts FP8 with 128x128 block scales, bf16 elsewhere) on GB300 (sm_103) is served by one modeling_v2 target, qwen4_exp/qwen3_8_flash_next_fp8__sm_103__unified, at every layout routing admits for it on one node. Outside models/qwen4_exp/ it touches only the target's tests, the accuracy references and the CI and QA test lists; nothing outside modeling_v2 qwen4_exp changes. The pre-merge GB300 multi-GPU stage runs only when a listed multi-GPU path changes, so a change to the target's csrc/ or ops/comm/ is run with /bot run --add-multi-gpu-test.

layout GPUs attention MoE
tp1 1 every head every expert
tp2 / tp4 2 / 4 TP whole-expert windows over the TP group (the FP8 128x128 blocks take no 320- or 160-row share)
tep2 / tep4 2 / 4 TP EP, a 256- / 128-expert window per rank
dep2 / dep4 2 / 4 attention DP (enable_attention_dp) EP

At those layouts the target runs CUDA graphs and the overlap scheduler; one-model MTP-Eagle with 1-7 draft tokens at the attention-TP layouts and at dep4, and the IndexShare draft path; KV cache auto, FP8 and NVFP4; a bf16 recurrent-state cache; chunked prefill; block reuse with recurrent-state snapshots; the breakable prefill CUDA graph under attention TP; disaggregated serving; and image and video input through the vision tower, each prefill chunk running the frames that hold its rows. Recurrent-state snapshots are taken at chunk ends; the in-forward snapshot split is not active (no stock hybrid manager plans it), and the target refuses at bind a manager that plans one without staging the PLE state.

routing.py (+25/-10 lines against main) sends a checkpoint the engine resolves as FP8_BLOCK_SCALES to the FP8 target at the NVFP4 target's layouts plus tp2 and tep2; every other checkpoint routes as before. As for the NVFP4 target, the target's own check_deployment declines what it does not run: another checkpoint shape, other MoE backends, a non-bf16 recurrent-state cache, MTP under attention DP at dep2, the breakable prefill graph under attention DP, and draft paths other than a linear one-model MTP-Eagle chain of 1-7.

Native code

The target's own CUDA kernels build into the qwen4_exp family library (models/qwen4_exp/csrc/, libth_modeling_v2_qwen4_exp.so, loaded only when routing selects the FP8 target; nothing is added under cpp/):

op what it does where it runs
fp8_block_moe_tpb TP-block decode MoE: routing, both expert GEMMs and the shared-expert slice in 8-CTA clusters, one launch per layer decode steps of at most 8 rows at tp4 / tep4 with TRTLLM_MODELING_V2_DECODE_MOE_TPB=rr (opt-in: the block-major expert copy costs about 31 GB a rank)
hc_allreduce_inject_norm the hyper-connection TP sum fused with the injection and grouped norm, one IPC Lamport launch on the stock all-reduce workspace decode and verify steps of 33-3071 rows at tp2 / tep2 / tp4 / tep4
fp8_block_scale_moe_front one-launch decode routing (renormalize top-10 of 512) and 1x128 FP8 quantization ahead of the trtllm-gen FP8 block-scale MoE runner attention-TP steps of 9-1024 rows

The prefill experts of chunks of 2048 rows and more use a target copy of the CuTe blockwise GEMM with a contiguous grouped mode and a fused 1x128 FP8 epilogue (ops/moe/blockwise_gemm_grouped.py).

Collectives

The TP sum and the attention-DP gather and scatter are the shared catalog/comm entries (allreduce, allgather, reducescatter), as the NVFP4 target calls them. The target's own collectives are the MNNVL plain TP sum for decode-sized calls, the fused TP sum + injection above, the Lamport gather of the split QSA selection, and the copy-engine pair exchange of tp2 / tep2 prefill. Under attention DP the lm_head stays replicated, as in the stock models.

Under attention DP the MoE round trip is the all-gather / reduce-scatter, as every modeling_v2 target runs it. TRTLLM_FORCE_COMM_METHOD=NVLINK_ONE_SIDED (the stock MoE communication override) runs it instead as an all-to-all dispatch / combine on the stock NVLink one-sided MoE all-to-all (NVLinkOneSided, the strategy the stock fused MoE selects first under attention DP), ops/comm/moe_a2a__dp_dispatch_combine.py: each rank routes its own rows, sends them with their top-k ids and weights to the ranks whose expert windows they use, runs its window over what it received, and gets its rows' partials back summed. It is a reference of that path for modeling_v2 targets, not a faster one here: a token's top-10 of 512 experts reaches 3.78 of 4 windows on average.

Shared operations

The target calls the catalog's cublas_mm, thop_attention and fused_qk_norm_rope and the family's causal_conv1d_update, gdn_decode_rows_gated_norm, gdn_prefill_conv_post and bf16_to_fp32 as they are; its shapes are TARGET_CELLS in those operations' tests. Its own ops/ holds only the operations it runs differently.

The shared qwen4_exp operations also gain the FP8 target's cells and defaulted parameters the FP8 target alone passes (prefill_query_tile and wide_split in qsa_paged_index_scores, row_order in qsa_topk_expand_decode, wide in qwen4_exp_hc_up_gate_mean, direct_launch in fused_sigmoid_mul_inplace and rms_norm_gated_token_major, pdl / inject_block in ple_gate_conv_inject_norm, conv in spec_rollback_commit). The NVFP4 target does not pass them, but these default-path changes do reach it; its outputs are unchanged (bitwise in the family operation tests, GSM8K and pass keys below):

operation change on the NVFP4 target's path
ple_ngram_embed per-sequence calls still launch BLOCK_P = next_power_of_2(max_rows), and its 1, 2 and 4 variants compile at a geometry's first call; both kernels take a row window (row_lo, row_hi, row_base, not specialized) that reads the whole table by default
qsa_paged_sparse_gqa_nvfp4 split-K and merge kernels: gdc_wait_with_memory_clobber() instead of tl.extra.cuda.gdc_wait()
qsa_paged_index_scores visible-split and request-group kernels: the same clobbered wait; the request-group kernel no longer specializes on rows or on the alignment of query_positions / request_indices; a single-row-tile visible-split launch compiles ROWS_PER_TILE at the query tile (one variant for every row count up to it); the request-group kernel compiles its three TILES_PER_PROGRAM variants at the first eager call of a geometry
qwen4_exp_hc_inject_norm compiles both num_warps variants (8 and 2) at the first eager call; the variant each call launches is unchanged
ple_gate_conv_inject_norm the conv kernels no longer specialize on row and sequence counts or index-vector alignment, and compile every MAX_P variant and the rows kernel at the first eager call
spec_rollback_commit the stage kernel takes two new constexprs (STAGE_CONV, PREV_IN_POOL); the NVFP4 call compiles the same arithmetic
qsa_topk_expand_decode the narrow kernel takes one more pointer argument (the wide kernel's row order), which it does not read
PLE host table (ple_host_table.py, chunk_broker.py, NVFP4 modeling.py / weights.py) the node key is a hash of sampled table rows instead of the checkpoint path; a load error releases the table; check_deployment reads TRTLLM_MODELING_V2_PLE_HOST_TABLE, so a misspelled value fails at routing; a requested mode that does not run is a warning

The clobbered waits are preventive: tl.extra.cuda.gdc_wait() is inline asm without a memory clobber, so nothing stops the compiler from hoisting a load of the predecessor's output above the wait. No NVFP4 failure was traced to it; the FP8 target's own split kernel read a stale flag for this reason in a 2-rank test. No NVFP4 performance was measured before and after these changes.

PLE host table

The PLE n-gram table (320,001,536 x 160 FP8 rows, 47.7 GiB) lives in host memory. qwen4_exp/ple_host_table.py cuts it into 8 fixed chunks and stores it per rank, per server or per node (TRTLLM_MODELING_V2_PLE_HOST_TABLE, below) for both qwen4_exp targets; qwen4_exp/chunk_broker.py hands a node's chunks between processes. It replaces the FP8 target's earlier shared host table and comes from #22077, whose two commits this PR now carries. ple_ngram_embed gains row_window=(lo, hi, base) for the server mode; its default reads the whole table, which is what the NVFP4 target passes under rank. The node table is keyed by a digest of the table's shape and of sampled rows of every chunk (table_identity); attaching to a published table checks a second, offset sample and falls back on a mismatch.

Switches

environment variable default effect
TRTLLM_MODELING_V2_DECODE_MOE_TPB off rr: TP-block decode MoE at tp4 / tep4 (memory for speed)
TRTLLM_MODELING_V2_PLE_HOST_TABLE rank how the 47.7 GiB PLE n-gram host table is stored (both qwen4_exp targets): rank one private copy a rank; server only this rank's row chunks, the gathers summed over the attention-TP group (exact; attention TP only); node one copy a node, CUDA VMM host chunks handed between processes with SCM_RIGHTS over abstract Unix sockets (no files, no ptrace), falling back to server or rank when a rank cannot take part
TRTLLM_MODELING_V2_ROWTILE_SCRATCH_GROUPS 0 1: context row-tile launches by request group past the 1 GiB FP8 scratch; outputs of those steps change by rounding
TRTLLM_FORCE_COMM_METHOD (stock) unset NVLINK_ONE_SIDED: under attention DP, the MoE round trip as the stock NVLink one-sided all-to-all instead of all-gather / reduce-scatter (needs MPI, attention DP and MNNVL); values other than ALLGATHER are declined at route time

Evidence for the switches that exist for speed (GB300):

switch or path evidence
TRTLLM_MODELING_V2_DECODE_MOE_TPB=rr (about 31 GB a rank) agg tep4 MTP3, concurrency 1, paired 900 s AgentX screens against the same engine without it (2 runs against a pool of 3): x +9.7 % (interval +5.4 to +13.9 %), median ITL +9.1 %, y +1.8 %, TTFT p90 flat; decode step at batch 1: 3.106 to 2.839 ms (same node, two rounds). GSM8K passed
TRTLLM_MODELING_V2_ROWTILE_SCRATCH_GROUPS=1 context steps with many requests over an FP8 pool: 13 to 20 % faster in a step microbenchmark. End to end not conclusive: single 3600 s runs on the disaggregated deployment below gave y +2.6 % at concurrency 208 and -4.9 % at 240 (different trees), and 900 s screens were inconclusive at 224 and regressed TTFT p90 at 192. It stays opt-in
contiguous grouped blockwise GEMM for prefill experts (ops/moe/blockwise_gemm_grouped.py) context step of 16k rows at 67k context: 195.4 to 180.6 ms (-7.6 %), 16k at 128k: -9.0 %, 8 x 5632: -9.2 % (same node, alternating off/on); the routed-expert region at 16k rows: 1115 to 693 us. Not bitwise to the trtllm-gen runner (errors against an fp32 reference of the same size)

Performance

InferenceX AgentX trace replay, official metric (x = 1000 / P90 full-response inter-token latency, y = total tokens/s per GPU over all of a deployment's GPUs), golden acceptance length per draft length (MTP2 2.06, MTP3 2.32), 3600 s windows, one deployment of 4 GB300s on an exclusive node, GSM8K (1319 questions, real acceptance) on each configuration. These runs predate this head: they used earlier revisions of this PR on older main commits, and two of them a draft-vocabulary subset this PR no longer has. They were not repeated on the current base.

deployment tree x y TTFT p90 GSM8K
agg tep4, MTP3, DECODE_MOE_TPB=rr, concurrency 1 main d1118ea + this PR, draft vocabulary on 597.9 5,839 0.51 s 95.03
same + #21930, #21935, #21931 (main 1a296dc), draft vocabulary on 609.5 6,008 0.48 s 94.96
disagg 1 ctx tep2 + 1 gen tep2, MTP2, block reuse per conversation, host KV tier, the PLE table shared per node (the removed PLE_SHARED_HOST=node, now PLE_HOST_TABLE=node), concurrency 240 main d1118ea + this PR, draft vocabulary on (gen) 120.2 207,508 24.15 s 95.49
same main 52cd3d0 + this PR, no draft vocabulary 115.9 229,128 18.85 s 95.19
same + #21930, #21935, #21931 (main 1a296dc), draft vocabulary on (gen), two runs 119.4 / 119.0 284,548 / 284,921 1.13 / 1.12 s 94.73

The agg tep4 point without the draft vocabulary on main 52cd3d0 has no valid run: both attempts compiled Triton kernels inside the timed window (_causal_conv1d_update_kernel, fixed by #21930, and the family _gdn_gates_kernel), which the benchmark audit rejects.

Which deployment needs which runtime PR:

PR used by evidence
#21931 (same-node direct peer KV copy) disaggregated at concurrency 272 on the full stack, turning it off: y 314,548 to 279,711 (-11.1 %), TTFT p90 1.30 to 13.43 s
#21935 (KV cache manager v2: host path, in-forward snapshot restore, cold-tier sizing, sub-agent demotion) disaggregated part of the full-stack runs above; no single-PR run
#21930 (runtime fixes; Triton kernels that compiled inside timed windows; opt-in early PDL release in the MNNVL one-shot all-reduce) both; the agg point cannot be measured without its causal_conv1d fix part of the full-stack runs above; the early PDL release measured -2.1 % decode step time at batch 1 (its own description)

With all three, the disaggregated point moves from y 207,508 to 284,548 and TTFT p90 from 24.15 to 1.13 s (both with the draft vocabulary), and the agg point from x 597.9 to 609.5.

Commits

commit scope
modeling_v2 qwen4_exp: family operations at the FP8 checkpoint's shapes family operations and their tests
modeling_v2 qwen4_exp: Qwen3.8-Flash-Next-FP8 target for TP1/TP2/TEP2/DEP2/TP4/TEP4/DEP4 on GB300 the target (Python and csrc), routing.py, the target's unit tests, the target's accuracy tests and GSM8K reference, CI lists
the [None][fix], [None][perf], [None][test] and [None][chore] commits after them review fixes: formatting, route-time declines, PDL and all-reduce settings, kernel barriers, the stock TRTLLM_FORCE_COMM_METHOD, IpcMemory, OpWrappers for every target op, family-op fixes, new accuracy tests and measured gates, the QA list, the tp2/tep2 HT replay precompile, the per-step row order of the wide decode selection, C++ guideline and input checks in csrc/, the vision tower per chunk, the FP8 KV range probe, cuBLAS for eager draft steps, and the later review rounds: the IPC setup agreement of the copy-engine exchange, the PLE node table's owner checks, leak-free failure paths and content key, all-reduce strategy and switch-scope declines, graphed-step low-M projections, split-K and GDN replay precompiles, the per-request vision tower cache, and the GDN replay history addressed by its own replay slots (main #21324); the last five commits keep every change inside qwen4_exp: the Lamport protocol copy in csrc/, the direct-launch copies pinned bitwise to the registered ops, the MNNVL / SYMM_MEM strategy mapping in the target
modeling_v2 qwen4_exp fp8: drop the shared host PLE table, modeling_v2 qwen4_exp: PLE host table per rank, per server or per node from #22077: the PLE host table for both qwen4_exp targets

Test Coverage

  • Unit, tests/unittest/_torch/modeling_v2/models/qwen4_exp/qwen3_8_flash_next_fp8__sm_103__unified/ (53 test functions): per file, one test_certified_cells over every (operation, cell) pair, against each operation's reference (the three native ops and their fake implementations included); one parametrized test per feature: routing admission and refusal, core and pool layout per rank, rank-local weight shards, speculative rollback, the in-forward snapshot, the vision tower against Hugging Face, the PLE host table modes and its chunk broker, the all-to-all round trip's wiring; and the 2- and 4-rank collectives under mpirun, the all-to-all dispatch / combine on NVLinkOneSided included. The family-operation tests under qwen4_exp/ops/ and the catalog tests of cublas_mm, thop_attention and fused_qk_norm_rope drive the FP8 target's cells as TARGET_CELLS.
  • Accuracy, tests/integration/defs/accuracy/modeling_v2/qwen4_exp/qwen3_8_flash_next_fp8__sm_103__unified/test_accuracy.py::TestModelingV2Qwen38FlashNextFp8Sm103Unified, three engines against the FP8_BLOCK_SCALES references: test_mtp3_gsm8k[tep4_fp8_kv_breakable] (attention TP with EP, three drafts, FP8 KV, block reuse with recurrent-state snapshots, chunked prefill, the breakable prefill graph), test_mtp3_gsm8k[dep4] (attention DP, three drafts, block reuse, chunked prefill) and test_disaggregated_dep2; test_mtp3_gsm8k[dep4_alltoall] runs the dep4 engine with the all-to-all round trip. test_mtp3_long_context (tep4, three drafts, pass keys over 3k-6k-token prompts with IndexShare: the sparse attention path end to end) runs pre-merge in l0_gb300.yml, test_tp1_gsm8k (one GPU, no speculation) in l0_b300.yml; acceptance length is gated against measured rows of acceptance_length.yaml (floor 95%), and the FP8-KV engine against its own measured GSM8K row. tp2 and tp4 run tep2's and tep4's expert windows; tp1 and tep2 are held by the unit tests' per-rank cases. CI: the unit tests pre-merge in l0_b300.yml (single GPU) and l0_cpu.yml (cpu_only); the collectives and test_mtp3_gsm8k[tep4_fp8_kv_breakable] pre-merge in l0_gb300.yml (4 GPUs); test_mtp3_gsm8k[dep4] and test_disaggregated_dep2 post-merge in l0_gb300_multi_gpus.yml, with measured .test_durations entries so the least-duration split places them (GB300-4_GPUs-PyTorch-Post-Merge-1 already runs 232 of its 240 minutes on main); test_mtp3_gsm8k[dep4_alltoall] is in the QA function list only.

At this head, on GB300 (4 GPUs), main 62a9192 (after modeling_v2 moved out of _torch/_experimental in #22286) + this PR with the native code rebuilt:

  • qwen3_8_flash_next_fp8__sm_103__unified/ (with comm/ and the checkpoint mounted): 761 passed; test_modeling_v2_contracts.py: 19 passed. The family fused_sigmoid_mul_inplace and rms_norm_gated_token_major tests (with the direct-launch pins): 36 and 22 passed.
  • TestModelingV2Qwen38FlashNextFp8Sm103Unified: test_mtp3_gsm8k[tep4_fp8_kv_breakable]: passed; on main c654904 test_tp1_gsm8k and [dep4], on main 3d3d542 [dep4_alltoall]: passed. Main [None][feat] Qwen3.5 GB300 round 2: decode/prefill kernels, KVCM2 remote tier, disagg GEN admission #21997 added a scale pointer to the stock sigmoid-mul kernel and row groups to the token-major gated norm; the family direct-launch copies follow both (the old sigmoid-mul copy failed the target's tests with a missing-argument TypeError), and their bitwise pins against the registered ops pass.
  • On main 2728049 (6f6b6af): CI L0 #70087 passed with the multi-GPU stages; the post-merge stages GB300-4_GPUs-PyTorch-Post-Merge-1/2/3 (L0 #70130) passed [dep4] (11.1 min), [dep4_alltoall] (14.0 min) and test_disaggregated_dep2 (6.0 min), but Post-Merge-1 hit its 240-minute walltime, which the durations entries and the QA-only [dep4_alltoall] address. On main 08b8c22: test_modeling_v2_thop_attention.py 159 passed, 2 skipped; test_mtp3_long_context (IndexShare): passed.
  • Main [TRTLLM-16303][refact] Keep hybrid speculative replay state in KV cache regions #21324 keeps the GDN replay history and its bookkeeping in replay slots of their own with 5-field work items; the target's verify path passes the generation rows' replay_state_indices to its CuTe replay, and the bind precompiles with the work-item width and the separate-slot flag as keys. Without it, test_mtp3_gsm8k[tep4_fp8_kv_breakable] fails at warmup on the replay's contiguity check.
  • On main 762a8dc: the family operation tests under ops/ 984 passed, the NVFP4 target's directory 568 passed and 2 skipped, the catalog all-reduce op matrix 2 passed and 1 skipped.
  • Earlier, on main 2722406: test_modeling_v2_routing.py, test_modeling_v2_contracts.py, test_modeling_v2_ci_selection.py 372 passed, 70 skipped (two ci_selection failures came from main and are fixed there by [None][fix] Move two Qwen4 unit tests under ops/ and read YAML QA lists in the Modeling V2 CI selection check #22085 and [https://nvbugs/7082290][infra] Waive 1 failed case(s) on CPU #22128); on main 0c6ac5f the stock test_fused_qk_norm_rope_gate.py and mamba/test_layernorm_gated.py 173 and 25 passed, TestModelingV2Qwen38FlashNextNvfp4Sm103Default::test_tp1_gsm8k passed.

At the previous head (before the round-3 review fixes):

  • TestModelingV2Qwen38FlashNextFp8Sm103Unified: test_mtp3_gsm8k[tep4_fp8_kv_breakable] 95.64 (AL 3.268), [dep4] 95.83 (AL 3.403), [dep4_alltoall] 94.69 (AL 3.396), test_mtp3_long_context passed (AL 4.000), test_tp1_gsm8k 94.54. TestModelingV2Qwen38FlashNextNvfp4Sm103Default::test_tp1_gsm8k 95.91.

Earlier, on GB300 (4 GPUs), with the native code rebuilt, at the revision of this PR rebased on main 52cd3d0 (before the review fixes):

  • qwen3_8_flash_next_fp8__sm_103__unified/ with comm/, the NVFP4 target's test_routing.py, test_modeling_v2_routing.py, test_modeling_v2_contracts.py and test_modeling_v2_ci_selection.py: 1079 passed, 70 skipped.
  • The catalog tests of fused_qk_norm_rope, thop_attention and cublas_mm: 215 passed, 3 skipped.
  • TestModelingV2Qwen38FlashNextFp8Sm103Unified::test_mtp3_gsm8k[tep4_fp8_kv_breakable], ::test_mtp3_gsm8k[dep4], ::test_mtp3_gsm8k[dep4_alltoall] and ::test_disaggregated_dep2: passed.
  • NVFP4 target (family operations changed): qwen3_8_flash_next_nvfp4__sm_103__default/ and the family operation tests under ops/: 1533 passed, 2 skipped; TestModelingV2Qwen38FlashNextNvfp4Sm103Default::test_mtp3_gsm8k_and_long_context and ::test_tp1_gsm8k: passed.
  • scripts/generate_llm_args_golden_manifest.py and scripts/generate_trtllm_serve_schemas.py: no change.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

@Wanli-Jiang
Wanli-Jiang force-pushed the user/williamj/qwen38fn-fp8-gb300-mv2 branch 2 times, most recently from 4c85226 to b66a7cc Compare November 9, 2026 05:08
@Wanli-Jiang Wanli-Jiang added the api-compatible Accepted LLM API contract change that is backwards-compatible label Nov 9, 2026
@Wanli-Jiang
Wanli-Jiang force-pushed the user/williamj/qwen38fn-fp8-gb300-mv2 branch from b66a7cc to b990c7b Compare November 9, 2026 13:50
@Wanli-Jiang Wanli-Jiang removed the api-compatible Accepted LLM API contract change that is backwards-compatible label Nov 9, 2026
@Wanli-Jiang
Wanli-Jiang force-pushed the user/williamj/qwen38fn-fp8-gb300-mv2 branch 4 times, most recently from 49bec84 to 0eb6909 Compare November 10, 2026 06:49
@Wanli-Jiang
Wanli-Jiang marked this pull request as ready for review November 10, 2026 06:52
@Wanli-Jiang
Wanli-Jiang requested review from a team as code owners November 10, 2026 06:52
@Wanli-Jiang
Wanli-Jiang force-pushed the user/williamj/qwen38fn-fp8-gb300-mv2 branch from 0eb6909 to ae763d5 Compare November 10, 2026 06:53
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • Resume reviews
  • Trigger review

Walkthrough

This change adds the qwen3_8_flash_next_fp8__sm_103__unified target for qwen4_exp, expands shared qwen4_exp routing and launch paths for FP8 layouts, adds target-local CUDA, Triton, and Python operators, and updates unit and integration coverage for the new FP8 path.

Changes

Unified FP8 target and shared qwen4_exp updates

Layer / File(s) Summary
Target registration and runtime foundation
tensorrt_llm/_torch/_experimental/modeling_v2/models/qwen4_exp/csrc/CMakeLists.txt, tensorrt_llm/_torch/_experimental/modeling_v2/models/qwen4_exp/qwen3_8_flash_next_fp8__sm_103__unified/..., tensorrt_llm/_torch/_experimental/modeling_v2/models/qwen4_exp/routing.py, tensorrt_llm/_torch/autotuner.py
Registers the SM103 FP8 unified target, adds package markers and native_ops loading, routes FP8 layouts through the new target, exposes the current capture context on AutoTuner, and adds pool and host-sharing helpers plus fake ops for the target package.
Attention kernels and dispatch
tensorrt_llm/_torch/_experimental/modeling_v2/models/qwen4_exp/ops/attention/..., tensorrt_llm/_torch/_experimental/modeling_v2/models/qwen4_exp/qwen3_8_flash_next_fp8__sm_103__unified/ops/attention/..., tensorrt_llm/_torch/_experimental/modeling_v2/models/qwen4_exp/qwen3_8_flash_next_fp8__sm_103__unified/vision.py
Adds or updates the attention decode, prefill, sparse GQA, top-k, speculative metadata, draft-row, paged cache, and vision-position operators and kernels for the FP8 target. The changes add new launch options, new kernel variants, and new reference and validation paths.
MoE, HC, and SSM operators
tensorrt_llm/_torch/_experimental/modeling_v2/models/qwen4_exp/qwen3_8_flash_next_fp8__sm_103__unified/ops/moe/..., tensorrt_llm/_torch/_experimental/modeling_v2/models/qwen4_exp/qwen3_8_flash_next_fp8__sm_103__unified/ops/hc/..., tensorrt_llm/_torch/_experimental/modeling_v2/models/qwen4_exp/qwen3_8_flash_next_fp8__sm_103__unified/ops/ssm/...
Adds the FP8 MoE front end, FP8 block-scale and TPB MoE paths, hyper-connection inject/norm and decode kernels, and SSM decode, convolution, and delta-rule operators. The shared qwen4_exp HC, MoE, and SSM implementations also gain new launch modes, wider kernel variants, and updated validation and references.
Communication operators
tensorrt_llm/_torch/_experimental/modeling_v2/models/qwen4_exp/qwen3_8_flash_next_fp8__sm_103__unified/ops/comm/...
Adds copy-engine exchange, Lamport all-gather, MNNVL TP sum, attention-DP MoE dispatch/combine, and fused inject-norm all-reduce operators plus their native kernels, wrappers, and support code.
Unit and integration coverage
tests/.../qwen4_exp/..., tests/integration/..., tests/integration/test_lists/...
Adds target-specific unit tests, communication matrix tests, vision tests, weight and routing tests, and accuracy entries and test-list wiring for the FP8 unified target. The tests cover eager paths, graph replay, capture cells, and distributed execution cases.

Estimated code review effort: 5 (Critical) | ~120 minutes

Change: Feature

Suggested reviewers: chzblych, erictsai-nv

Security architecture risk: Moderate · up to d7ac3

The change introduces shared resources across local serving processes. When optional sharing uses a directory writable by other local users, those users can interfere with startup and publication. Existing ownership checks and deployment restrictions limit exposure, but do not fully contain this availability risk.

Retained concerns

  • Medium · security · inferred: Optional host sharing crosses a local-user availability boundary without containing publication failure. In a shared writable sticky directory, another unprivileged local user who can observe the rendezvous stem and owner PID can pre-create the predictable publication temporary path. The owner's unlink then fails with a permission error, which propagates out of weight loading before publication and lock release. Pending processes have no local wait deadline and can remain blocked until the owner closes or exits. This is conditional on sharing being enabled and the attacker accessing the same rendezvous directory; a service-private directory avoids this attack path.
Security review details

Security Blast Radius

  • inferred — The supported availability exposure is host-local and conditional: it affects sharing-enabled serving processes using a rendezvous directory accessible to an untrusted local user. The inspected path does not establish remote-only exploitation, cross-node propagation, credential access, or privilege escalation.

Security Findings and Attack Paths

  • inferred — An attacker able to create entries in the same shared sticky directory can occupy the PID-qualified temporary publication name before the owner publishes. Foreign ownership prevents the owner's unlink, aborting publication. Ownership and no-follow checks prevent trusting that entry, but the failure is not converted to private storage or bounded waiter recovery.

Trust Boundaries and Controls

  • observed — Allocation attachment relies on operating-system file-descriptor access and UID-scoped rendezvous files, not a distinct tenant identity. CUDA attachment requests a read-only mapping but retries with read-write access after a RuntimeError. Therefore read-only table usage is not an unconditional hardware-enforced isolation guarantee; whether all sharing processes belong to one trusted domain is deployment context not supplied here.

Hardening Proposals

  • proposed — Use a service-private rendezvous namespace for sharing and define bounded recovery for publication failure. Explicitly document whether every attached process is trusted to modify the shared allocation; where that trust is not intended, require enforced read-only attachment rather than a writable fallback.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title follows the required [ticket][type] format and clearly identifies the FP8 target and supported layouts.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The PR description is thorough and largely complete. It explains the target scope and the supported layouts, lists the native CUDA ops with their use, documents the environment switches and shared-ope…
Full details: Description check

Explanation

The PR description is thorough and largely complete. It explains the target scope and the supported layouts, lists the native CUDA ops with their use, documents the environment switches and shared-operation changes, and gives detailed test coverage and performance evidence. Its structure differs from the repository template: the template's '## Description' and '## Test Coverage' headings are present, but the PR checklist is included and checked. The PR title is not part of this check. The description also says the performance numbers predate the current head, and it does not state which API-compatibility label applies. The description is mostly complete, so this check passes.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@Wanli-Jiang

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #83699 [ run ] triggered by Bot. Commit: 63ea228 Link to invocation

@yuxianq

yuxianq commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

@farazkh80, could you help to review this PR as trt-llm-torch-attention-devs? Thanks~

Assignment explanation
Componentskernels/sparse (35%); kernels/linear (20%); runtime/metadata-workspaces (10%); runtime/sparse-cache (8%); runtime/kv-cache/layout-io (7%); runtime/parallelism (7%); integration/attention-tests (5%); unmapped (8%)
Tagsalgorithm:sparse (65%); algorithm:linear (35%); specific-algorithm:qsa (65%); specific-algorithm:gated-delta (35%); representation:gqa (100%); phase:prefill (40%); phase:decode (60%); features:quantization (25%); features:speculative-decoding (35%); features:paged-kv (25%); features:cuda-graphs (15%)
PR effort13 points — The existing-family references support an FP8 execution-target feature rather than a new architecture. Sparse kernel scheduling, selection semantics, recurrent verification, rollback and TP exchange warrant thirteen attention-review points independently of MoE size. This is provisional because most implementation bodies are absent.
ImportanceImportant — Visible added contracts cover sparse row-tile and split-K attention, BF16 recurrent history/checkpoint updates, and accepted-count corrections to positions and KV lengths. These are sensitive numerical, cache-state and speculative-execution changes.
Affinity (0–1)0.399 component/tag match; 0.299 after PIC preference
Outstanding points at selection29 → 42 (+13)
Points already reserved on this PR0
Effective weight1
Completed review points (preceding 168 hours)16
Combined points after assignment58
Demand-derived target45.00 points (soft share)
Additional fair share before assignment0.00 points
Share explanationWhole PR / qualification or coverage constraint exceeds a soft share; assignment is allowed.
Selection ruleaffinity with 0.3-share guard
Assignment scope (farazkh80)all

Affinity and workload are snapshots at selection, not live counters. Existing reservations on this PR are counted once.

@yuxianq
yuxianq requested review from farazkh80 and removed request for QiJune November 10, 2026 07:21
…y history by its replay slots

The stock hybrid cache manager can keep the GDN replay history and its
bookkeeping in replay slots of their own (the mamba metadata's
replay_state_indices) and hands out five-field work items. The FP8 target's
bind checks the bookkeeping and history against the replay slots, its verify
steps (decode and mixed) pass the generation rows' replay slots to the CuTe
replay, and the replay reads the history, its bookkeeping and the work items
at their own slot and element strides. The precompile keys carry the work-item
width and whether the replay slots are separate, so the bind compiles exactly
the kernels a step launches.

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…ies pinned to the registered ops

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…ithout a stock accessor

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…r MNNVL and SYMM_MEM in the FP8 target

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
… the FP8 target's collectives

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…otocol in the FP8 target

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…ests in the GB300 shards

Record measured durations for test_mtp3_gsm8k[dep4] and test_disaggregated_dep2 so the least-duration split places them, and keep test_mtp3_gsm8k[dep4_alltoall] (the opt-in all-to-all path) in the QA list only.

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…m and MoE all-to-all collective matrices

Both ops are thin calls of stock kernels (the MNNVL all-reduce without fusion, NVLinkOneSided dispatch / combine); the matrices kept are the target's own protocols: the fused HC TP sum, the Lamport index-split gather and the copy-engine exchange.

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…d-norm launches in the family direct-launch copies

The sigmoid-mul kernel now takes a scale pointer before the strides, and the token-major gated norm launch tiles large row counts in groups with contiguity and evenness hints; the qwen4_exp copies launch the same way, which their bitwise pins against the registered ops check.

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…_sum and MoE all-to-all collective matrices

Both op entries declare certified cells, which the modeling_v2 contract requires a test to drive; the matrices are those tests, and MNNVL tp_sum is on the default decode path at tp2 and tp4.

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…e path after the modeling_v2 move

Main moved modeling_v2 out of _torch/_experimental; ruff re-sorts the shortened import paths in the FP8 target's and the family's files, and the hc sync test builds its package path without the old directory.

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
@Wanli-Jiang
Wanli-Jiang force-pushed the user/williamj/qwen38fn-fp8-gb300-mv2 branch from 8d5c189 to 5969704 Compare November 17, 2026 08:16
@Wanli-Jiang

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@Wanli-Jiang
Wanli-Jiang enabled auto-merge (squash) November 17, 2026 08:24
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85437 [ run ] triggered by Bot. Commit: 5969704 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85437 [ run ] completed with state SUCCESS. Commit: 5969704
/LLM/main/L0_MergeRequest_PR pipeline #70722 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

… before its answer threads on close

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
@Wanli-Jiang

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85478 [ run ] triggered by Bot. Commit: 2fa681c Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85478 [ run ] completed with state SUCCESS. Commit: 2fa681c
/LLM/main/L0_MergeRequest_PR pipeline #70749 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@Wanli-Jiang

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85503 [ run ] triggered by Bot. Commit: 2fa681c Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85503 [ run ] completed with state SUCCESS. Commit: 2fa681c
/LLM/main/L0_MergeRequest_PR pipeline #70773 completed with status: 'SUCCESS'

CI Report

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci: full pre-merge approved ci: post-merge approved Approved by TRT-LLM CI approvers for broad post-merge CI requests

Projects

None yet

Development

Successfully merging this pull request may close these issues.