Repository navigation
[TRTLLM-16622][perf] DeepSeek-V4.1: CSA2 small-M decode with keys-as-M direct decode, rotated-back output and projection prefetch - #22368
Draft
ziyixiong-nv wants to merge 4 commits into
Conversation
This was referenced Nov 16, 2026
Merged
ziyixiong-nv
force-pushed
the
dsv41-mb/pr13-csa2-small-m-decode
branch
from
November 17, 2026 11:35
c10aed6 to
6dde04f
Compare
ziyixiong-nv
force-pushed
the
dsv41-mb/pr13-csa2-small-m-decode
branch
2 times, most recently
from
November 18, 2026 09:32
749f8b4 to
e5e349d
Compare
…, PDL asm ordering, mHC boundary alignment and constant zeros
Five fixes; each applies wherever its code runs, for every model:
- fusedCatFp4 / fusedCatFp8 (FP4 / FP8 index queries of the DSA indexers): the output offsets
were 32-bit ints, so from 2^25 / 2^24 rows (tokens x index heads, e.g. 512K / 256K tokens at 64
heads) they overflowed and the kernels stored outside their output. The offsets are 64-bit.
- CSA2 glue kernels: tensor1d/tensor2d took CuTe DSL's default unit stride, which some DSL builds
type as 32-bit, so element offsets of 2^31 and more (2 GiB of a byte tensor) wrapped. The unit
stride is Int64.
- CuTe DSL griddepcontrol.wait / launch_dependents had no memory clobber, so LLVM could move
ordinary global loads (e.g. .nc loads of a predecessor's output) above the PDL wait and read the
output before the predecessor finished. Both asm statements carry "~{memory}", as CUTLASS C++.
- The mHC all-reduce boundary is compiled for 16-byte-aligned operands; a contiguous view whose
storage offset is not a multiple of 16 bytes (an un-finalized MoE output can be one) faulted on
misaligned vector accesses. Such operands are copied to aligned storage first.
- CSA2 staging asked for a zero constant of each step's row count and kept one tensor per count it
had seen, so a long-running eager context server kept growing its HBM use. Those per-row zeros
now come from the shared buffer pool (get_memory_buffers(), reserved while a CUDA graph is
captured) and are zeroed per call.
Not measured separately. With the memory clobber, a GB300 decode step at attention TP4 (6 decode
rows, CUDA graphs) is unchanged: 3.796 ms vs 3.801 / 3.815 ms. The pooled zeros cost one small
fill per staging launch without a main mapping.
Test: test_mhc_allreduce_boundary_matches_allreduce_then_boundary also feeds the MoE-finalize
boundary contiguous views at a one-element storage offset, eager and in CUDA-graph replays, and
compares with the all-reduce then the boundary bit for bit.
Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…C1-only TRTLLM-Gen runs, a CuTe DSL FC2, one-cluster routing and dynamic EP DeepSeek-V4.1 decode at attention TP spends its MoE time in small TRTLLM-Gen calls whose FC2 cubins run in rounds and whose routing sits on the critical path. This change builds on the FC1-only TRTLLM-Gen run and the bit-identical CuTe DSL FC2 (moe_lowm_fc2, decode_moe) of NVIDIA#21997: - MoEDecodeOptions.cute_fc2_rows lets a model take that CuTe FC2 alone, without the decode tuning and engine: un-finalized attention-TP calls of up to that many rows on SM100/SM103, also on the runner's routing tables (FC1-only run_moe_routing_tables_fc1 and run_moe_bounded_fc1). - moe_lowm_fc2 re-applies the +0 of a cubin's zero-filled K tail where it has no tail of its own, so it also stands in for the FC2 cubins with 512-column K tiles. - Routes up to 8 tokens (384 experts, top-6) in one 8-CTA cluster that also lays out TRTLLM-Gen's routing tables with one warp and releases the expert GEMM right after its wait. - Splits each call's used experts home-first over the two EP ranks on batch-1 workers (dynamic EP); both ranks hold their MoE-TP shard of all routed experts, and the GEMMs launch with the dynamic tables' CTA bound (run_moe_bounded) instead of the worst case. ModelConfig carries the executor's max_batch_size for this (filled by the model loader; no LLM API change). - Serves the router GEMM as K-chunk partials from 12 rows per rank instead of 65 on SM100/SM103 for a model that opts in; every other caller keeps 65. Gating: DeepSeek-V4.1 declares MoEDecodeOptions(cute_fc2_rows=16, one_cluster_route=True) and sets extra_attrs "moe_dynamic_ep" (which also needs attention TP, MoE EP2 with MoE TP >= 4, batch-1 workers, sqrt-softplus routing, TRTLLM-Gen NVFP4 and no EPLB or DWDP, on SM100/SM103) and "moe_router_partials_small_m". The one-cluster route kernel takes the 384-expert top-6 tables: bit-identical to the CuTe route kernel's up to the row order in an expert's block. Qwen3.5 and every other model, DeepSeek-V4 included, keep their kernels and outputs, except that the shared CuTe FC2 now adds a 512-column K tile cubin's +0 tail where the intermediate size is an odd multiple of 256 (an exactly zero output takes the cubin's sign); SM107 is unchanged. The MXFP4 x MXFP8 runner is unchanged: its CuTe FC2 is left to NVIDIA#22371, which needs it for the DSpark draft's MXFP4 experts. Numerics: bit-identical, except that dynamic EP changes which rows fall into each rank's BF16 partial (FP32 summation order, like an EPLB placement change): the all-reduced MoE output stays within 8.7e-3 of the call's largest value of the static split's. Measured on GB300, DeepSeek-V4.1 decode at TEP8 batch 1 with DSpark (k=5): - CuTe FC2 with the one-cluster route kernel: decode step 2.803 / 2.777 -> 2.670 ms. - Dynamic EP: MoE half of a layer -1.44 us (-58 us per step); EP-rank imbalance 2.8 -> 0.7 us. - Early GEMM release and one-warp tables: -10..-30 us and -11 us per step. - CTA-bounded GEMMs: FC1 grid 324 -> 162 CTAs at 6 rows; FC2 no longer waits for FC1's end. Router partials from 12 rows, DEP32 at 48 rows per rank: step 7.06 -> 7.00 ms (-0.9%). Tests: test_moe_lowm_fc2.py adds the opt-in on the one-cluster route kernel's tables against the FC2 cubin (eager and CUDA graph replay) and a 768-column shape; test_one_cluster_route_matches_the_cute_kernel; test_moe_ep_dynamic_matches_static_split (4 GPUs, dynamic against static split, both route kernels, finalized and un-finalized, CUDA graph replay); test_megamoe_e2e gains the 48-row shape for the router partials. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…e attention O-B in the boundary launch At attention TP the decode mHC boundaries all-reduce the sublayer's partial output in their own launch (one-shot Lamport protocol). On SM100/SM103, every change bitwise: - ObAllReduceBoundaryKernel runs the attention's small-M MXFP8 O-B projection, its all-reduce and the FFN-entry boundary as two roles of one grid: O-B CTAs compute 8-column tiles as the small-M GEMM does and push them into every rank's slots; token CTAs poll, sum in rank order and finish the boundary. It engages for up to 16 rows when the FFN boundary is split and all-reduces in-kernel and the O-B shard tiles the grid (attention TP8); the decoder layer asks the attention for its O-B input through defer_ob_to_boundary, and defer_ob_projection builds the ObUnprojected. - The CTA counted last advances the Lamport state (a hand-over warp in the cluster form of the all-reduce boundary), so no CTA spins on the arrival counter after its work. - A rank takes its own slot from the words it pushed instead of waiting for their multicast copy. - The 4-CTA cluster form (up to 12 rows) exchanges RMSNorm partials with st.async on an mbarrier; the MoE-finalize rows load right after the routing. - Behind a captured CuTe FC2 the MNNVL MoE boundary reads its routing and clears the previous buffer before its dependency wait. The MoE boundary that enters a layer whose O-B is deferred prefetches that O-B weight into L2. GB300, attention TP8 batch-1 decode with DSpark k=5, per decode step: last-CTA hand-over in the O-B boundary -6..-10 us; routing before the wait, own slot and hand-over warp -16 us together (FC2 end -> MoE boundary end -0.38 / -0.43 us per layer); the O-B weight prefetch together with the CSA2 O-A releasing the deferred boundary after its wait about -29 us. The fused O-B launch and the mbarrier exchange are not measured separately (the exchange with the early MoE rows: 3.688 -> 3.684 ms per step at attention TP4 with 6 rows, within the run-to-run spread). Test: test_mhc_ob_boundary_matches_ob_then_boundary (1, 6 and 16 rows, with and without the NVFP4 copy, MNNVL; 50 graph replays between a MoE-finalize boundary and a library all-reduce on one workspace) equals the O-B GEMM then the boundary byte for byte; test_mhc_moe_boundary_behind_lowm_fc2 (2 GPUs, MNNVL) replays captured layers of attention boundary, FC1-only MoE, CuTe FC2 and MoE boundary (with the O-B prefetch) with the eager bits; test_mhc_allreduce_boundary_matches_allreduce_then_boundary also covers the 6-row MoE input. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…M direct decode, rotated-back output and projection prefetch Small CSA2 decode steps (attention TP and small attention-DP batches) on SM100/SM103 now: - run the direct sparse decode at 8 query heads per rank as keys-as-M attention: a CTA's key rows are the MMA's M and the heads its N, with one softmax over the CTA's keys and the partial outputs sent to their channel owners by st.async. The row decode pairs its FP32 scale multiplies (same products), resolves the key rows first and overlaps the merge transfers; - write the decode output rotated back for the small-M O-A, which then skips its inverse RoPE on load (the O-A latent is bit for bit the same); - release a SWA / Reuse layer's decode right after the row write's own dependency wait, load a Reuse layer's main rows before the decode's wait, and launch a Full layer's decode with plain edges once its selections are final; - pack the index queries in the index-query projection and publish the layer's cache rows on a side stream, off the index chain, and refresh a captured decode forward's SWA slots on a side stream from the first layer; - prefetch the O-A / O-B weights into L2 beside the fused decode at up to 64 rows, run the FP8 Q-B as one 128x256 tile per CTA up to 64 rows, take two O-A tiles per CTA at <= 8 rows, run the query projections at the decode's shared-memory carveout, and release a deferred O-B boundary after the small-M O-A's own wait. Outputs are bit-identical except the keys-as-M attention's FP32 summation order: max abs 2-4e-3 against the two-part launch, both as close to an FP32 reference. GB300 measurements: - Keys-as-M decode, attention TP8 x one request with 5 draft tokens: decode step 2.802 / 2.809 -> 2.726 ms (-76..-83 us); GSM8K 96.66 strict / 96.82 flexible against 96.74 / 96.74. - Rotated-back output with the key resolve first: about -39 us per step (same setup); Full layers' decode behind their final selections: -35 us per step. - FP8 Q-B tile, attention DP32 at 48 rows per rank: step 7.087 -> 6.996 ms (-1.3%); the O-A/O-B prefetch at 48 rows per rank: -0.044..-0.070 ms per step. Tests: test_direct_decode.py (keys-as-M against the two-part launch and an FP32 reference, the rotated-back output at 8 heads against the O-A's rotation on load) and test_module.py (packed index queries with side-stream row publication), eagerly and in CUDA graph replays. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
ziyixiong-nv
force-pushed
the
dsv41-mb/pr13-csa2-small-m-decode
branch
from
November 18, 2026 16:28
e5e349d to
6f6f4cb
Compare
Open
1 task
1 task done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
Summary
Small CSA2 decode steps of DeepSeek-V4.1 (attention TP, and attention DP up to 64 rows) on
SM100/SM103: a keys-as-M direct sparse decode at 8 query heads per rank, a decode output already
rotated back for the small-M O-A, earlier releases between the row write, the decode and the
output projection, index queries packed by their projection with the cache rows published off the
index chain, and L2 prefetch and tiling of the projections.
Changes
csa2/direct_decode.py):CTA's key rows are the MMA's M and the heads its N, one softmax over the CTA's keys, and the
partial outputs go to their channel owners with
st.asyncon mbarriers; key rows are resolvedfirst, and the (max, sum) pairs and P.V halves are sent as each finishes;
output_rope), as the small-MO-A rotates its input on load, so that O-A runs without inverse RoPE;
mul.rn.f32x2, same products);main_readywithout a final query loads a Reuse layer's main rows before the dependent wait andconverts them after it;
plain_edgelaunches a Full layer's decode once its selections arefinal while it still releases its dependents early.
csa2/backend.py,csa2/kernel.py,csa2/quantization.py,csa2/cache_manager.py): a SWA / Reuse layer's row write releases the direct decode right afterits own dependency wait (
release_after_wait);publish_layer_rowsfactors out the rowpublication.
csa2/module.py,csa2/params.py,csa2/projection.py,csa2/query_projection.py):the index-query projection (
mxfp8_lowm_index_query) and publishes the cache rows on a sidestream after Q-A + KV, off the index chain;
(
kernels/blackwell/csa2_weight_prefetch.py, new: one-warp CTAs on the SMs the decode leavesfree, each with a 24 us issue budget);
at <= 8 rows; the query projections run at the keys-as-M decode's shared-memory carveout; the
small-M O-A releases an O-B projection deferred to the all-reduce boundary after its own wait.
csa2/metadata.py,models/modeling_deepseekv41.py): a captured attention-TP decodeforward refreshes the SWA slots on a side stream from its first CSA2 layer; readers and the end of
the model forward join it.
modules/low_m_gemm.py:mxfp8_lowm_linear(smem_carveout=)sets a nonzero preferredshared-memory carveout on the compiled small-M kernel; the query projections pass the keys-as-M
decode's.
Performance
GB300 (SM103), DeepSeek-V4.1. "TEP8" is attention TP8 with MoE TP4 x EP2, serving one request with
5 draft tokens (6 decode rows).
Reuse layers 8.93 -> 7.08 us, index layers 9.3 -> 7.7 us.
instead of 14.4-14.9 us; -35 us per step across the Full layers' attention halves.
decode's wait: -30..-35 us per step together with [TRTLLM-16622][perf] DeepSeek-V4.1: mHC split boundaries and coefficient kernels for decode batches #22360's one-launch mHC coefficients (attention
TP8, two nodes).
coefficient kernel's carveout ([TRTLLM-16622][perf] DeepSeek-V4.1: mHC split boundaries and coefficient kernels for decode batches #22360). O-A releasing a deferred O-B boundary after its wait: about
-29 us per step together with [TRTLLM-16622][perf] DeepSeek-V4.1: mHC all-reduce boundaries with the attention O-B in the boundary launch #22367's O-B weight prefetch.
(3.672 -> 3.655 ms).
6.996 ms (-1.3%). O-A / O-B prefetch at 48 rows per rank: -0.044..-0.070 ms per step.
(-0.39 us per Reuse-layer launch at TP4), the SWA refresh fork (the first layer's cache write ends
2.6-4.2 us earlier) and the O-A two-tile CTAs (measured together with [TRTLLM-16622][perf] DeepSeek-V4.1: small-M FP8/MXFP8 GEMM with BF16 weight conversion, swapped operands, packed index queries and SwiGLU output #22359's swapped operands).
Compatibility
launches. Attention-TP-only paths check
mapping.enable_attention_dp.another FP32 order: max abs difference 2-4e-3 to the two-part launch, both equally close to an FP32
reference. GSM8K (1,319 questions, greedy, TEP8 with real draft acceptance): 96.66 strict / 96.82
flexible, acceptance length 4.011, against 96.74 / 96.74 and 3.999.
modules/low_m_gemm.pysets the carveout only when a caller passes a nonzerosmem_carveout(CSA2 only);DeepseekV41Model.forwardjoins the SWA refresh fork. Other modelsare unaffected.
Dependencies
Stack on #22367, which stacks on #22354 and #22363:
lowm_tiles,mxfp8_lowm_index_query, and the small-M GEMMs'n_tilesandrelease_after_waitarguments;defer_ob_projectionanddefer_ob_to_boundary(the small-M O-A hands O-B to theall-reduce boundary).
Checked: cherry-picks cleanly onto main + #22354 + #22363 + #22367, and the branch merges cleanly with
#22358.
Test Coverage
test_direct_decode.py:test_keys_as_m_attention_matches_the_two_part_launch(page-mapped main and SWAselections; FP32 reference and the two-part launch; empty rows output zeros; the Reuse launch bit for bit; graph
replays) and
test_output_rotated_back_gives_the_o_a_latent_bitwise(8 heads: the O-A latent bit for bit,eagerly and in graph replays). Both run at 8 heads per rank, attention TP8, which no E2E test runs.
test_module.py:test_index_query_pack_publishes_rows_beside_index_chain(the packing GEMMagainst the cache write's packing at 1/6/12/16 rows with zero, tiny, huge, NaN and inf rows; cache
bytes, selections and output of a Full layer, eagerly and replayed). Two existing tests follow the new signatures.
attention-TP4 decode step of the GB300 DeepSeek-V4.1 GSM8K test.
test_swa_and_reuse_rows_release_direct_decode_after_wait,test_swa_refresh_fork.py(whole file), the SWA-onlyand slot cases of the keys-as-M test and the 16-head case of the rotated-back test. main's
l0_b300list runs thewhole
unittest/_torch/attention/sparse/csa2directory.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.