Repository navigation
[None][feat] Qwen3.8-Flash-Next FP8 modeling_v2: one target for TP1/TP2/TEP2/DEP2/TP4/TEP4/DEP4 on GB300 - #21621
Conversation
4c85226 to
b66a7cc
Compare
b66a7cc to
b990c7b
Compare
49bec84 to
0eb6909
Compare
0eb6909 to
ae763d5
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #83699 [ run ] triggered by Bot. Commit: |
|
@farazkh80, could you help to review this PR as trt-llm-torch-attention-devs? Thanks~ Assignment explanation
Affinity and workload are snapshots at selection, not live counters. Existing reservations on this PR are counted once. |
…y history by its replay slots The stock hybrid cache manager can keep the GDN replay history and its bookkeeping in replay slots of their own (the mamba metadata's replay_state_indices) and hands out five-field work items. The FP8 target's bind checks the bookkeeping and history against the replay slots, its verify steps (decode and mixed) pass the generation rows' replay slots to the CuTe replay, and the replay reads the history, its bookkeeping and the work items at their own slot and element strides. The precompile keys carry the work-item width and whether the replay slots are separate, so the bind compiles exactly the kernels a step launches. Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…ies pinned to the registered ops Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…ithout a stock accessor Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…r MNNVL and SYMM_MEM in the FP8 target Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
… the FP8 target's collectives Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…otocol in the FP8 target Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…ests in the GB300 shards Record measured durations for test_mtp3_gsm8k[dep4] and test_disaggregated_dep2 so the least-duration split places them, and keep test_mtp3_gsm8k[dep4_alltoall] (the opt-in all-to-all path) in the QA list only. Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…m and MoE all-to-all collective matrices Both ops are thin calls of stock kernels (the MNNVL all-reduce without fusion, NVLinkOneSided dispatch / combine); the matrices kept are the target's own protocols: the fused HC TP sum, the Lamport index-split gather and the copy-engine exchange. Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…d-norm launches in the family direct-launch copies The sigmoid-mul kernel now takes a scale pointer before the strides, and the token-major gated norm launch tiles large row counts in groups with contiguity and evenness hints; the qwen4_exp copies launch the same way, which their bitwise pins against the registered ops check. Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…_sum and MoE all-to-all collective matrices Both op entries declare certified cells, which the modeling_v2 contract requires a test to drive; the matrices are those tests, and MNNVL tp_sum is on the default decode path at tp2 and tp4. Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
…e path after the modeling_v2 move Main moved modeling_v2 out of _torch/_experimental; ruff re-sorts the shortened import paths in the FP8 target's and the family's files, and the hc sync test builds its package path without the old directory. Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
8d5c189 to
5969704
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #85437 [ run ] triggered by Bot. Commit: |
|
PR_Github #85437 [ run ] completed with state
|
… before its answer threads on close Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #85478 [ run ] triggered by Bot. Commit: |
|
PR_Github #85478 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #85503 [ run ] triggered by Bot. Commit: |
|
PR_Github #85503 [ run ] completed with state |
Description
Qwen3.8-Flash-Next-FP8 (
Qwen4ExpForConditionalGeneration; routed experts FP8 with 128x128 block scales, bf16 elsewhere) on GB300 (sm_103) is served by one modeling_v2 target,qwen4_exp/qwen3_8_flash_next_fp8__sm_103__unified, at every layout routing admits for it on one node. Outsidemodels/qwen4_exp/it touches only the target's tests, the accuracy references and the CI and QA test lists; nothing outside modeling_v2 qwen4_exp changes. The pre-merge GB300 multi-GPU stage runs only when a listed multi-GPU path changes, so a change to the target'scsrc/orops/comm/is run with/bot run --add-multi-gpu-test.enable_attention_dp)At those layouts the target runs CUDA graphs and the overlap scheduler; one-model MTP-Eagle with 1-7 draft tokens at the attention-TP layouts and at dep4, and the IndexShare draft path; KV cache auto, FP8 and NVFP4; a bf16 recurrent-state cache; chunked prefill; block reuse with recurrent-state snapshots; the breakable prefill CUDA graph under attention TP; disaggregated serving; and image and video input through the vision tower, each prefill chunk running the frames that hold its rows. Recurrent-state snapshots are taken at chunk ends; the in-forward snapshot split is not active (no stock hybrid manager plans it), and the target refuses at bind a manager that plans one without staging the PLE state.
routing.py(+25/-10 lines against main) sends a checkpoint the engine resolves asFP8_BLOCK_SCALESto the FP8 target at the NVFP4 target's layouts plus tp2 and tep2; every other checkpoint routes as before. As for the NVFP4 target, the target's owncheck_deploymentdeclines what it does not run: another checkpoint shape, other MoE backends, a non-bf16 recurrent-state cache, MTP under attention DP at dep2, the breakable prefill graph under attention DP, and draft paths other than a linear one-model MTP-Eagle chain of 1-7.Native code
The target's own CUDA kernels build into the qwen4_exp family library (
models/qwen4_exp/csrc/,libth_modeling_v2_qwen4_exp.so, loaded only when routing selects the FP8 target; nothing is added undercpp/):fp8_block_moe_tpbTRTLLM_MODELING_V2_DECODE_MOE_TPB=rr(opt-in: the block-major expert copy costs about 31 GB a rank)hc_allreduce_inject_normfp8_block_scale_moe_frontThe prefill experts of chunks of 2048 rows and more use a target copy of the CuTe blockwise GEMM with a contiguous grouped mode and a fused 1x128 FP8 epilogue (
ops/moe/blockwise_gemm_grouped.py).Collectives
The TP sum and the attention-DP gather and scatter are the shared
catalog/commentries (allreduce,allgather,reducescatter), as the NVFP4 target calls them. The target's own collectives are the MNNVL plain TP sum for decode-sized calls, the fused TP sum + injection above, the Lamport gather of the split QSA selection, and the copy-engine pair exchange of tp2 / tep2 prefill. Under attention DP thelm_headstays replicated, as in the stock models.Under attention DP the MoE round trip is the all-gather / reduce-scatter, as every modeling_v2 target runs it.
TRTLLM_FORCE_COMM_METHOD=NVLINK_ONE_SIDED(the stock MoE communication override) runs it instead as an all-to-all dispatch / combine on the stock NVLink one-sided MoE all-to-all (NVLinkOneSided, the strategy the stock fused MoE selects first under attention DP),ops/comm/moe_a2a__dp_dispatch_combine.py: each rank routes its own rows, sends them with their top-k ids and weights to the ranks whose expert windows they use, runs its window over what it received, and gets its rows' partials back summed. It is a reference of that path for modeling_v2 targets, not a faster one here: a token's top-10 of 512 experts reaches 3.78 of 4 windows on average.Shared operations
The target calls the catalog's
cublas_mm,thop_attentionandfused_qk_norm_ropeand the family'scausal_conv1d_update,gdn_decode_rows_gated_norm,gdn_prefill_conv_postandbf16_to_fp32as they are; its shapes areTARGET_CELLSin those operations' tests. Its ownops/holds only the operations it runs differently.The shared qwen4_exp operations also gain the FP8 target's cells and defaulted parameters the FP8 target alone passes (
prefill_query_tileandwide_splitinqsa_paged_index_scores,row_orderinqsa_topk_expand_decode,wideinqwen4_exp_hc_up_gate_mean,direct_launchinfused_sigmoid_mul_inplaceandrms_norm_gated_token_major,pdl/inject_blockinple_gate_conv_inject_norm,convinspec_rollback_commit). The NVFP4 target does not pass them, but these default-path changes do reach it; its outputs are unchanged (bitwise in the family operation tests, GSM8K and pass keys below):ple_ngram_embedBLOCK_P = next_power_of_2(max_rows), and its 1, 2 and 4 variants compile at a geometry's first call; both kernels take a row window (row_lo,row_hi,row_base, not specialized) that reads the whole table by defaultqsa_paged_sparse_gqa_nvfp4gdc_wait_with_memory_clobber()instead oftl.extra.cuda.gdc_wait()qsa_paged_index_scoresrowsor on the alignment ofquery_positions/request_indices; a single-row-tile visible-split launch compilesROWS_PER_TILEat the query tile (one variant for every row count up to it); the request-group kernel compiles its threeTILES_PER_PROGRAMvariants at the first eager call of a geometryqwen4_exp_hc_inject_normnum_warpsvariants (8 and 2) at the first eager call; the variant each call launches is unchangedple_gate_conv_inject_normMAX_Pvariant and the rows kernel at the first eager callspec_rollback_commitSTAGE_CONV,PREV_IN_POOL); the NVFP4 call compiles the same arithmeticqsa_topk_expand_decodeple_host_table.py,chunk_broker.py, NVFP4modeling.py/weights.py)nodekey is a hash of sampled table rows instead of the checkpoint path; a load error releases the table;check_deploymentreadsTRTLLM_MODELING_V2_PLE_HOST_TABLE, so a misspelled value fails at routing; a requested mode that does not run is a warningThe clobbered waits are preventive:
tl.extra.cuda.gdc_wait()is inline asm without a memory clobber, so nothing stops the compiler from hoisting a load of the predecessor's output above the wait. No NVFP4 failure was traced to it; the FP8 target's own split kernel read a stale flag for this reason in a 2-rank test. No NVFP4 performance was measured before and after these changes.PLE host table
The PLE n-gram table (320,001,536 x 160 FP8 rows, 47.7 GiB) lives in host memory.
qwen4_exp/ple_host_table.pycuts it into 8 fixed chunks and stores it per rank, per server or per node (TRTLLM_MODELING_V2_PLE_HOST_TABLE, below) for both qwen4_exp targets;qwen4_exp/chunk_broker.pyhands a node's chunks between processes. It replaces the FP8 target's earlier shared host table and comes from #22077, whose two commits this PR now carries.ple_ngram_embedgainsrow_window=(lo, hi, base)for theservermode; its default reads the whole table, which is what the NVFP4 target passes underrank. Thenodetable is keyed by a digest of the table's shape and of sampled rows of every chunk (table_identity); attaching to a published table checks a second, offset sample and falls back on a mismatch.Switches
TRTLLM_MODELING_V2_DECODE_MOE_TPBrr: TP-block decode MoE at tp4 / tep4 (memory for speed)TRTLLM_MODELING_V2_PLE_HOST_TABLErankrankone private copy a rank;serveronly this rank's row chunks, the gathers summed over the attention-TP group (exact; attention TP only);nodeone copy a node, CUDA VMM host chunks handed between processes withSCM_RIGHTSover abstract Unix sockets (no files, no ptrace), falling back toserverorrankwhen a rank cannot take partTRTLLM_MODELING_V2_ROWTILE_SCRATCH_GROUPSTRTLLM_FORCE_COMM_METHOD(stock)NVLINK_ONE_SIDED: under attention DP, the MoE round trip as the stock NVLink one-sided all-to-all instead of all-gather / reduce-scatter (needs MPI, attention DP and MNNVL); values other thanALLGATHERare declined at route timeEvidence for the switches that exist for speed (GB300):
TRTLLM_MODELING_V2_DECODE_MOE_TPB=rr(about 31 GB a rank)TRTLLM_MODELING_V2_ROWTILE_SCRATCH_GROUPS=1ops/moe/blockwise_gemm_grouped.py)Performance
InferenceX AgentX trace replay, official metric (x = 1000 / P90 full-response inter-token latency, y = total tokens/s per GPU over all of a deployment's GPUs), golden acceptance length per draft length (MTP2 2.06, MTP3 2.32), 3600 s windows, one deployment of 4 GB300s on an exclusive node, GSM8K (1319 questions, real acceptance) on each configuration. These runs predate this head: they used earlier revisions of this PR on older main commits, and two of them a draft-vocabulary subset this PR no longer has. They were not repeated on the current base.
DECODE_MOE_TPB=rr, concurrency 1PLE_SHARED_HOST=node, nowPLE_HOST_TABLE=node), concurrency 240The agg tep4 point without the draft vocabulary on main 52cd3d0 has no valid run: both attempts compiled Triton kernels inside the timed window (
_causal_conv1d_update_kernel, fixed by #21930, and the family_gdn_gates_kernel), which the benchmark audit rejects.Which deployment needs which runtime PR:
With all three, the disaggregated point moves from y 207,508 to 284,548 and TTFT p90 from 24.15 to 1.13 s (both with the draft vocabulary), and the agg point from x 597.9 to 609.5.
Commits
modeling_v2 qwen4_exp: family operations at the FP8 checkpoint's shapesmodeling_v2 qwen4_exp: Qwen3.8-Flash-Next-FP8 target for TP1/TP2/TEP2/DEP2/TP4/TEP4/DEP4 on GB300csrc),routing.py, the target's unit tests, the target's accuracy tests and GSM8K reference, CI lists[None][fix],[None][perf],[None][test]and[None][chore]commits after themTRTLLM_FORCE_COMM_METHOD,IpcMemory,OpWrappers for every target op, family-op fixes, new accuracy tests and measured gates, the QA list, the tp2/tep2 HT replay precompile, the per-step row order of the wide decode selection, C++ guideline and input checks incsrc/, the vision tower per chunk, the FP8 KV range probe, cuBLAS for eager draft steps, and the later review rounds: the IPC setup agreement of the copy-engine exchange, the PLEnodetable's owner checks, leak-free failure paths and content key, all-reduce strategy and switch-scope declines, graphed-step low-M projections, split-K and GDN replay precompiles, the per-request vision tower cache, and the GDN replay history addressed by its own replay slots (main #21324); the last five commits keep every change inside qwen4_exp: the Lamport protocol copy incsrc/, the direct-launch copies pinned bitwise to the registered ops, the MNNVL / SYMM_MEM strategy mapping in the targetmodeling_v2 qwen4_exp fp8: drop the shared host PLE table,modeling_v2 qwen4_exp: PLE host table per rank, per server or per nodeTest Coverage
tests/unittest/_torch/modeling_v2/models/qwen4_exp/qwen3_8_flash_next_fp8__sm_103__unified/(53 test functions): per file, onetest_certified_cellsover every (operation, cell) pair, against each operation's reference (the three native ops and their fake implementations included); one parametrized test per feature: routing admission and refusal, core and pool layout per rank, rank-local weight shards, speculative rollback, the in-forward snapshot, the vision tower against Hugging Face, the PLE host table modes and its chunk broker, the all-to-all round trip's wiring; and the 2- and 4-rank collectives under mpirun, the all-to-all dispatch / combine onNVLinkOneSidedincluded. The family-operation tests underqwen4_exp/ops/and the catalog tests ofcublas_mm,thop_attentionandfused_qk_norm_ropedrive the FP8 target's cells asTARGET_CELLS.tests/integration/defs/accuracy/modeling_v2/qwen4_exp/qwen3_8_flash_next_fp8__sm_103__unified/test_accuracy.py::TestModelingV2Qwen38FlashNextFp8Sm103Unified, three engines against theFP8_BLOCK_SCALESreferences:test_mtp3_gsm8k[tep4_fp8_kv_breakable](attention TP with EP, three drafts, FP8 KV, block reuse with recurrent-state snapshots, chunked prefill, the breakable prefill graph),test_mtp3_gsm8k[dep4](attention DP, three drafts, block reuse, chunked prefill) andtest_disaggregated_dep2;test_mtp3_gsm8k[dep4_alltoall]runs the dep4 engine with the all-to-all round trip.test_mtp3_long_context(tep4, three drafts, pass keys over 3k-6k-token prompts with IndexShare: the sparse attention path end to end) runs pre-merge inl0_gb300.yml,test_tp1_gsm8k(one GPU, no speculation) inl0_b300.yml; acceptance length is gated against measured rows ofacceptance_length.yaml(floor 95%), and the FP8-KV engine against its own measured GSM8K row. tp2 and tp4 run tep2's and tep4's expert windows; tp1 and tep2 are held by the unit tests' per-rank cases. CI: the unit tests pre-merge inl0_b300.yml(single GPU) andl0_cpu.yml(cpu_only); the collectives andtest_mtp3_gsm8k[tep4_fp8_kv_breakable]pre-merge inl0_gb300.yml(4 GPUs);test_mtp3_gsm8k[dep4]andtest_disaggregated_dep2post-merge inl0_gb300_multi_gpus.yml, with measured.test_durationsentries so the least-duration split places them (GB300-4_GPUs-PyTorch-Post-Merge-1 already runs 232 of its 240 minutes on main);test_mtp3_gsm8k[dep4_alltoall]is in the QA function list only.At this head, on GB300 (4 GPUs), main 62a9192 (after modeling_v2 moved out of
_torch/_experimentalin #22286) + this PR with the native code rebuilt:qwen3_8_flash_next_fp8__sm_103__unified/(withcomm/and the checkpoint mounted): 761 passed;test_modeling_v2_contracts.py: 19 passed. The familyfused_sigmoid_mul_inplaceandrms_norm_gated_token_majortests (with the direct-launch pins): 36 and 22 passed.TestModelingV2Qwen38FlashNextFp8Sm103Unified:test_mtp3_gsm8k[tep4_fp8_kv_breakable]: passed; on main c654904test_tp1_gsm8kand[dep4], on main 3d3d542[dep4_alltoall]: passed. Main [None][feat] Qwen3.5 GB300 round 2: decode/prefill kernels, KVCM2 remote tier, disagg GEN admission #21997 added a scale pointer to the stock sigmoid-mul kernel and row groups to the token-major gated norm; the family direct-launch copies follow both (the old sigmoid-mul copy failed the target's tests with a missing-argument TypeError), and their bitwise pins against the registered ops pass.[dep4](11.1 min),[dep4_alltoall](14.0 min) andtest_disaggregated_dep2(6.0 min), but Post-Merge-1 hit its 240-minute walltime, which the durations entries and the QA-only[dep4_alltoall]address. On main 08b8c22:test_modeling_v2_thop_attention.py159 passed, 2 skipped;test_mtp3_long_context(IndexShare): passed.replay_state_indicesto its CuTe replay, and the bind precompiles with the work-item width and the separate-slot flag as keys. Without it,test_mtp3_gsm8k[tep4_fp8_kv_breakable]fails at warmup on the replay's contiguity check.ops/984 passed, the NVFP4 target's directory 568 passed and 2 skipped, thecatalogall-reduce op matrix 2 passed and 1 skipped.test_modeling_v2_routing.py,test_modeling_v2_contracts.py,test_modeling_v2_ci_selection.py372 passed, 70 skipped (twoci_selectionfailures came from main and are fixed there by [None][fix] Move two Qwen4 unit tests under ops/ and read YAML QA lists in the Modeling V2 CI selection check #22085 and [https://nvbugs/7082290][infra] Waive 1 failed case(s) on CPU #22128); on main 0c6ac5f the stocktest_fused_qk_norm_rope_gate.pyandmamba/test_layernorm_gated.py173 and 25 passed,TestModelingV2Qwen38FlashNextNvfp4Sm103Default::test_tp1_gsm8kpassed.At the previous head (before the round-3 review fixes):
TestModelingV2Qwen38FlashNextFp8Sm103Unified:test_mtp3_gsm8k[tep4_fp8_kv_breakable]95.64 (AL 3.268),[dep4]95.83 (AL 3.403),[dep4_alltoall]94.69 (AL 3.396),test_mtp3_long_contextpassed (AL 4.000),test_tp1_gsm8k94.54.TestModelingV2Qwen38FlashNextNvfp4Sm103Default::test_tp1_gsm8k95.91.Earlier, on GB300 (4 GPUs), with the native code rebuilt, at the revision of this PR rebased on main 52cd3d0 (before the review fixes):
qwen3_8_flash_next_fp8__sm_103__unified/withcomm/, the NVFP4 target'stest_routing.py,test_modeling_v2_routing.py,test_modeling_v2_contracts.pyandtest_modeling_v2_ci_selection.py: 1079 passed, 70 skipped.fused_qk_norm_rope,thop_attentionandcublas_mm: 215 passed, 3 skipped.TestModelingV2Qwen38FlashNextFp8Sm103Unified::test_mtp3_gsm8k[tep4_fp8_kv_breakable],::test_mtp3_gsm8k[dep4],::test_mtp3_gsm8k[dep4_alltoall]and::test_disaggregated_dep2: passed.qwen3_8_flash_next_nvfp4__sm_103__default/and the family operation tests underops/: 1533 passed, 2 skipped;TestModelingV2Qwen38FlashNextNvfp4Sm103Default::test_mtp3_gsm8k_and_long_contextand::test_tp1_gsm8k: passed.scripts/generate_llm_args_golden_manifest.pyandscripts/generate_trtllm_serve_schemas.py: no change.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.