Skip to content

[TRTLLM-16622][perf] DeepSeek-V4.1: CSA2 small-M decode with keys-as-M direct decode, rotated-back output and projection prefetch - #22368

Draft
ziyixiong-nv wants to merge 4 commits into
NVIDIA:mainfrom
ziyixiong-nv:dsv41-mb/pr13-csa2-small-m-decode
Draft

ziyixiong-nv wants to merge 4 commits into
NVIDIA:mainfrom
ziyixiong-nv:dsv41-mb/pr13-csa2-small-m-decode

Conversation

@ziyixiong-nv

@ziyixiong-nv ziyixiong-nv commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

@coderabbitai summary

Description

Stacked draft. This branch carries #22354, #22363 and #22367 below its own commit. Review only the top commit; the branch will be rebased onto main as those PRs merge.

Summary

Small CSA2 decode steps of DeepSeek-V4.1 (attention TP, and attention DP up to 64 rows) on
SM100/SM103: a keys-as-M direct sparse decode at 8 query heads per rank, a decode output already
rotated back for the small-M O-A, earlier releases between the row write, the decode and the
output projection, index queries packed by their projection with the cache rows published off the
index chain, and L2 prefetch and tiling of the projections.

Changes

  • Direct decode (csa2/direct_decode.py):
    • keys-as-M attention at 8 query heads per rank (8-CTA clusters, at most 128 key rows per CTA): a
      CTA's key rows are the MMA's M and the heads its N, one softmax over the CTA's keys, and the
      partial outputs go to their channel owners with st.async on mbarriers; key rows are resolved
      first, and the (max, sum) pairs and P.V halves are sent as each finishes;
    • the output can be written rotated back by the query's RoPE angle (output_rope), as the small-M
      O-A rotates its input on load, so that O-A runs without inverse RoPE;
    • the row decode pairs its FP32 scale multiplies (mul.rn.f32x2, same products);
    • main_ready without a final query loads a Reuse layer's main rows before the dependent wait and
      converts them after it; plain_edge launches a Full layer's decode once its selections are
      final while it still releases its dependents early.
  • Backend and cache writes (csa2/backend.py, csa2/kernel.py, csa2/quantization.py,
    csa2/cache_manager.py): a SWA / Reuse layer's row write releases the direct decode right after
    its own dependency wait (release_after_wait); publish_layer_rows factors out the row
    publication.
  • Module (csa2/module.py, csa2/params.py, csa2/projection.py, csa2/query_projection.py):
    • a small direct-decode step of a Full or Reindex layer at attention TP packs its index queries in
      the index-query projection (mxfp8_lowm_index_query) and publishes the cache rows on a side
      stream after Q-A + KV, off the index chain;
    • O-A / O-B weights are prefetched into L2 beside the fused decode at up to 64 rows
      (kernels/blackwell/csa2_weight_prefetch.py, new: one-warp CTAs on the SMs the decode leaves
      free, each with a 24 us issue budget);
    • the FP8 Q-B runs one 128x256 tile per CTA up to 64 rows; the decode O-A takes two tiles per CTA
      at <= 8 rows; the query projections run at the keys-as-M decode's shared-memory carveout; the
      small-M O-A releases an O-B projection deferred to the all-reduce boundary after its own wait.
  • Metadata (csa2/metadata.py, models/modeling_deepseekv41.py): a captured attention-TP decode
    forward refreshes the SWA slots on a side stream from its first CSA2 layer; readers and the end of
    the model forward join it.
  • modules/low_m_gemm.py: mxfp8_lowm_linear(smem_carveout=) sets a nonzero preferred
    shared-memory carveout on the compiled small-M kernel; the query projections pass the keys-as-M
    decode's.

Performance

GB300 (SM103), DeepSeek-V4.1. "TEP8" is attention TP8 with MoE TP4 x EP2, serving one request with
5 draft tokens (6 decode rows).

Compatibility

  • Gating: CSA2 only (DeepSeek-V4.1); the new device paths run on SM100/SM103 and SM107 keeps its
    launches. Attention-TP-only paths check mapping.enable_attention_dp.
  • Numerics: bit-identical to the existing paths except the keys-as-M attention, which sums in
    another FP32 order: max abs difference 2-4e-3 to the two-part launch, both equally close to an FP32
    reference. GSM8K (1,319 questions, greedy, TEP8 with real draft acceptance): 96.66 strict / 96.82
    flexible, acceptance length 4.011, against 96.74 / 96.74 and 3.999.
  • Shared code: modules/low_m_gemm.py sets the carveout only when a caller passes a nonzero
    smem_carveout (CSA2 only); DeepseekV41Model.forward joins the SWA refresh fork. Other models
    are unaffected.

Dependencies

Stack on #22367, which stacks on #22354 and #22363:

Checked: cherry-picks cleanly onto main + #22354 + #22363 + #22367, and the branch merges cleanly with
#22358.

Test Coverage

  • test_direct_decode.py: test_keys_as_m_attention_matches_the_two_part_launch (page-mapped main and SWA
    selections; FP32 reference and the two-part launch; empty rows output zeros; the Reuse launch bit for bit; graph
    replays) and test_output_rotated_back_gives_the_o_a_latent_bitwise (8 heads: the O-A latent bit for bit,
    eagerly and in graph replays). Both run at 8 heads per rank, attention TP8, which no E2E test runs.
  • test_module.py: test_index_query_pack_publishes_rows_beside_index_chain (the packing GEMM
    against the cache write's packing at 1/6/12/16 rows with zero, tiny, huge, NaN and inf rows; cache
    bytes, selections and output of a Full layer, eagerly and replayed). Two existing tests follow the new signatures.
  • The early-released SWA / Reuse row writes and the forked SWA slot refresh have no unit test: they run on every
    attention-TP4 decode step of the GB300 DeepSeek-V4.1 GSM8K test.
  • Test functions: 3 new (one case each) and 2 adapted, +269 / -6 lines. Trimmed:
    test_swa_and_reuse_rows_release_direct_decode_after_wait, test_swa_refresh_fork.py (whole file), the SWA-only
    and slot cases of the keys-as-M test and the 16-head case of the rotated-back test. main's l0_b300 list runs the
    whole unittest/_torch/attention/sparse/csa2 directory.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

@ziyixiong-nv
ziyixiong-nv force-pushed the dsv41-mb/pr13-csa2-small-m-decode branch from c10aed6 to 6dde04f Compare November 17, 2026 11:35
@ziyixiong-nv
ziyixiong-nv force-pushed the dsv41-mb/pr13-csa2-small-m-decode branch 2 times, most recently from 749f8b4 to e5e349d Compare November 18, 2026 09:32
…, PDL asm ordering, mHC boundary alignment and constant zeros

Five fixes; each applies wherever its code runs, for every model:

- fusedCatFp4 / fusedCatFp8 (FP4 / FP8 index queries of the DSA indexers): the output offsets
  were 32-bit ints, so from 2^25 / 2^24 rows (tokens x index heads, e.g. 512K / 256K tokens at 64
  heads) they overflowed and the kernels stored outside their output. The offsets are 64-bit.
- CSA2 glue kernels: tensor1d/tensor2d took CuTe DSL's default unit stride, which some DSL builds
  type as 32-bit, so element offsets of 2^31 and more (2 GiB of a byte tensor) wrapped. The unit
  stride is Int64.
- CuTe DSL griddepcontrol.wait / launch_dependents had no memory clobber, so LLVM could move
  ordinary global loads (e.g. .nc loads of a predecessor's output) above the PDL wait and read the
  output before the predecessor finished. Both asm statements carry "~{memory}", as CUTLASS C++.
- The mHC all-reduce boundary is compiled for 16-byte-aligned operands; a contiguous view whose
  storage offset is not a multiple of 16 bytes (an un-finalized MoE output can be one) faulted on
  misaligned vector accesses. Such operands are copied to aligned storage first.
- CSA2 staging asked for a zero constant of each step's row count and kept one tensor per count it
  had seen, so a long-running eager context server kept growing its HBM use. Those per-row zeros
  now come from the shared buffer pool (get_memory_buffers(), reserved while a CUDA graph is
  captured) and are zeroed per call.

Not measured separately. With the memory clobber, a GB300 decode step at attention TP4 (6 decode
rows, CUDA graphs) is unchanged: 3.796 ms vs 3.801 / 3.815 ms. The pooled zeros cost one small
fill per staging launch without a main mapping.

Test: test_mhc_allreduce_boundary_matches_allreduce_then_boundary also feeds the MoE-finalize
boundary contiguous views at a one-element storage offset, eager and in CUDA-graph replays, and
compares with the all-reduce then the boundary bit for bit.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…C1-only TRTLLM-Gen runs, a CuTe DSL FC2, one-cluster routing and dynamic EP

DeepSeek-V4.1 decode at attention TP spends its MoE time in small TRTLLM-Gen calls whose FC2 cubins
run in rounds and whose routing sits on the critical path. This change builds on the FC1-only
TRTLLM-Gen run and the bit-identical CuTe DSL FC2 (moe_lowm_fc2, decode_moe) of NVIDIA#21997:

- MoEDecodeOptions.cute_fc2_rows lets a model take that CuTe FC2 alone, without the decode tuning
  and engine: un-finalized attention-TP calls of up to that many rows on SM100/SM103, also on the
  runner's routing tables (FC1-only run_moe_routing_tables_fc1 and run_moe_bounded_fc1).
- moe_lowm_fc2 re-applies the +0 of a cubin's zero-filled K tail where it has no tail of its own,
  so it also stands in for the FC2 cubins with 512-column K tiles.
- Routes up to 8 tokens (384 experts, top-6) in one 8-CTA cluster that also lays out TRTLLM-Gen's
  routing tables with one warp and releases the expert GEMM right after its wait.
- Splits each call's used experts home-first over the two EP ranks on batch-1 workers (dynamic EP);
  both ranks hold their MoE-TP shard of all routed experts, and the GEMMs launch with the dynamic
  tables' CTA bound (run_moe_bounded) instead of the worst case. ModelConfig carries the executor's
  max_batch_size for this (filled by the model loader; no LLM API change).
- Serves the router GEMM as K-chunk partials from 12 rows per rank instead of 65 on SM100/SM103
  for a model that opts in; every other caller keeps 65.

Gating: DeepSeek-V4.1 declares MoEDecodeOptions(cute_fc2_rows=16, one_cluster_route=True) and
sets extra_attrs "moe_dynamic_ep" (which also needs attention TP, MoE EP2 with MoE TP >= 4, batch-1
workers, sqrt-softplus routing, TRTLLM-Gen NVFP4 and no EPLB or DWDP, on SM100/SM103) and
"moe_router_partials_small_m". The one-cluster route kernel takes the 384-expert top-6 tables:
bit-identical to the CuTe route kernel's up to the row order in an expert's block. Qwen3.5 and every
other model, DeepSeek-V4 included, keep their kernels and outputs, except that the shared CuTe FC2
now adds a 512-column K tile cubin's +0 tail where the intermediate size is an odd multiple of 256
(an exactly zero output takes the cubin's sign); SM107 is unchanged. The MXFP4 x MXFP8 runner is
unchanged: its CuTe FC2 is left to NVIDIA#22371, which needs it for the DSpark draft's MXFP4 experts.

Numerics: bit-identical, except that dynamic EP changes which rows fall into each rank's BF16
partial (FP32 summation order, like an EPLB placement change): the all-reduced MoE output stays
within 8.7e-3 of the call's largest value of the static split's.

Measured on GB300, DeepSeek-V4.1 decode at TEP8 batch 1 with DSpark (k=5):
- CuTe FC2 with the one-cluster route kernel: decode step 2.803 / 2.777 -> 2.670 ms.
- Dynamic EP: MoE half of a layer -1.44 us (-58 us per step); EP-rank imbalance 2.8 -> 0.7 us.
- Early GEMM release and one-warp tables: -10..-30 us and -11 us per step.
- CTA-bounded GEMMs: FC1 grid 324 -> 162 CTAs at 6 rows; FC2 no longer waits for FC1's end.
Router partials from 12 rows, DEP32 at 48 rows per rank: step 7.06 -> 7.00 ms (-0.9%).

Tests: test_moe_lowm_fc2.py adds the opt-in on the one-cluster route kernel's tables against the
FC2 cubin (eager and CUDA graph replay) and a 768-column shape;
test_one_cluster_route_matches_the_cute_kernel; test_moe_ep_dynamic_matches_static_split (4 GPUs,
dynamic against static split, both route kernels, finalized and un-finalized, CUDA graph replay);
test_megamoe_e2e gains the 48-row shape for the router partials.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…e attention O-B in the boundary launch

At attention TP the decode mHC boundaries all-reduce the sublayer's partial output in their own
launch (one-shot Lamport protocol). On SM100/SM103, every change bitwise:
- ObAllReduceBoundaryKernel runs the attention's small-M MXFP8 O-B projection, its all-reduce and
  the FFN-entry boundary as two roles of one grid: O-B CTAs compute 8-column tiles as the small-M
  GEMM does and push them into every rank's slots; token CTAs poll, sum in rank order and finish
  the boundary. It engages for up to 16 rows when the FFN boundary is split and all-reduces
  in-kernel and the O-B shard tiles the grid (attention TP8); the decoder layer asks the attention
  for its O-B input through defer_ob_to_boundary, and defer_ob_projection builds the ObUnprojected.
- The CTA counted last advances the Lamport state (a hand-over warp in the cluster form of the
  all-reduce boundary), so no CTA spins on the arrival counter after its work.
- A rank takes its own slot from the words it pushed instead of waiting for their multicast copy.
- The 4-CTA cluster form (up to 12 rows) exchanges RMSNorm partials with st.async on an mbarrier;
  the MoE-finalize rows load right after the routing.
- Behind a captured CuTe FC2 the MNNVL MoE boundary reads its routing and clears the previous
  buffer before its dependency wait. The MoE boundary that enters a layer whose O-B is deferred
  prefetches that O-B weight into L2.

GB300, attention TP8 batch-1 decode with DSpark k=5, per decode step: last-CTA hand-over in the
O-B boundary -6..-10 us; routing before the wait, own slot and hand-over warp -16 us together
(FC2 end -> MoE boundary end -0.38 / -0.43 us per layer); the O-B weight prefetch together with the
CSA2 O-A releasing the deferred boundary after its wait about -29 us. The fused O-B launch and the
mbarrier exchange are not measured separately (the exchange with the early MoE rows: 3.688 ->
3.684 ms per step at attention TP4 with 6 rows, within the run-to-run spread).

Test: test_mhc_ob_boundary_matches_ob_then_boundary (1, 6 and 16 rows, with and without the NVFP4
copy, MNNVL; 50 graph replays between a MoE-finalize boundary and a library all-reduce on one
workspace) equals the O-B GEMM then the boundary byte for byte;
test_mhc_moe_boundary_behind_lowm_fc2 (2 GPUs, MNNVL) replays captured layers of attention
boundary, FC1-only MoE, CuTe FC2 and MoE boundary (with the O-B prefetch) with the eager bits;
test_mhc_allreduce_boundary_matches_allreduce_then_boundary also covers the 6-row MoE input.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…M direct decode, rotated-back output and projection prefetch

Small CSA2 decode steps (attention TP and small attention-DP batches) on SM100/SM103 now:
- run the direct sparse decode at 8 query heads per rank as keys-as-M attention: a CTA's key
  rows are the MMA's M and the heads its N, with one softmax over the CTA's keys and the
  partial outputs sent to their channel owners by st.async. The row decode pairs its FP32 scale
  multiplies (same products), resolves the key rows first and overlaps the merge transfers;
- write the decode output rotated back for the small-M O-A, which then skips its inverse RoPE on
  load (the O-A latent is bit for bit the same);
- release a SWA / Reuse layer's decode right after the row write's own dependency wait, load a
  Reuse layer's main rows before the decode's wait, and launch a Full layer's decode with plain
  edges once its selections are final;
- pack the index queries in the index-query projection and publish the layer's cache rows on a
  side stream, off the index chain, and refresh a captured decode forward's SWA slots on a side
  stream from the first layer;
- prefetch the O-A / O-B weights into L2 beside the fused decode at up to 64 rows, run the FP8 Q-B
  as one 128x256 tile per CTA up to 64 rows, take two O-A tiles per CTA at <= 8 rows, run the
  query projections at the decode's shared-memory carveout, and release a deferred O-B boundary
  after the small-M O-A's own wait.

Outputs are bit-identical except the keys-as-M attention's FP32 summation order: max abs 2-4e-3
against the two-part launch, both as close to an FP32 reference.

GB300 measurements:
- Keys-as-M decode, attention TP8 x one request with 5 draft tokens: decode step 2.802 / 2.809
  -> 2.726 ms (-76..-83 us); GSM8K 96.66 strict / 96.82 flexible against 96.74 / 96.74.
- Rotated-back output with the key resolve first: about -39 us per step (same setup); Full
  layers' decode behind their final selections: -35 us per step.
- FP8 Q-B tile, attention DP32 at 48 rows per rank: step 7.087 -> 6.996 ms (-1.3%); the O-A/O-B
  prefetch at 48 rows per rank: -0.044..-0.070 ms per step.

Tests: test_direct_decode.py (keys-as-M against the two-part launch and an FP32 reference, the
rotated-back output at 8 heads against the O-A's rotation on load) and test_module.py (packed index
queries with side-stream row publication), eagerly and in CUDA graph replays.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant