Repository navigation
[None][feat] specDSA: MXFP4 indexer K cache (DS-V4-Flash-FP4) and tail attention for 9 to 26 rows - #22678
Draft
longcheng-nv wants to merge 16 commits into
Draft
[None][feat] specDSA: MXFP4 indexer K cache (DS-V4-Flash-FP4) and tail attention for 9 to 26 rows#22678longcheng-nv wants to merge 16 commits into
longcheng-nv wants to merge 16 commits into
Conversation
Add tensorrt_llm/_torch/attention/kernels/blackwell/dsv4_sparse_decode, a CuTe DSL kernel that runs the DeepSeek-V4-Flash CSA decode step (FP8 indexer logits, exact top-512, KV index mapping and sparse MLA with the attention sink) as one cooperative launch over the 148 SMs of a B200 for 1, 2, 4 or 8 rows, with its unit test. The kernel is not wired into the model yet. Made-with: Claude Code (Opus 5.5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…stage release The stream math warps read each K stage's per-token scales, and at batch > 1 the row's weights from the Q / W stage, and then release the stage with an mbarrier arrive. The arrive does not order the warp's outstanding shared-memory loads before the TMA refill (async proxy), so a delayed load could read the next tile's data. Add fence.proxy.async.shared::cta before each of these releases (5 sites). - Without the fences no failure was seen in 668,002 checked launches. With extra co-resident load / store traffic (stress-only builds), the K-stage race changed the top-512 in up to 99 % of launches. - With the fences, 0 of 378,000 mixed-length stress launches failed, 252,000 of them under that traffic. The gates (252 real cases per batch size) and the unit test pass. - Cost (nsys, clean L2): +0.31 / +0.02 / +0.02 / +0.30 us at batch 1 / 2 / 4 / 8. Made-with: Claude Code (Opus 5.5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…, batched candidate publish at batch 8 - Batch > 1: the stream CTAs' TMA warps take row 0's descriptor from the kernel-entry load and prefetch the tensor maps at entry, both before the CTA barrier; at batch 2 the team CTA also loads the row and window lengths before its barrier setup. - Batch 8: the candidate publish issues its histogram atomics in batches of two per thread, and the last row issues its first batch before the count barrier. This relies on the previous commit's fences. Fused-kernel time against the previous commit (nsys, clean L2): -0.12 / -0.04 / -1.13 us at batch 2 / 4 / 8 (a second GPU: -0.27 / -1.18 us at batch 2 / 8); batch 1 is unchanged. The gates (252 real cases per batch size) and the unit test pass, and 0 of 136,500 stress launches (mixed-length and hot-row cases) failed. Made-with: Claude Code (Opus 5.5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…1 to 48 The megakernel served 1, 2, 4 and 8 rows. It now serves every batch size from 1 to 48. - 3, 5, 6 and 7 rows use the 4- and 8-row configurations. - 9 to 16 rows: the logits of a row live in a ring of 8 shared-memory slots. A slot is reused after its row's selection reached K_DONE, which the T poller (warp 12) or the publishers observe before the row that needs the slot. - 17 to 32 rows: a 32-entry per-row service state past the stream layout, slot-free barrier phases (r >> 3) - 1 & 1, up to 16 passes per team. - 33 to 48 rows: one generation word and five state words per row, and the compressed-KV block-table slices of the stream CTAs as a ring of 32 entries, each reusable entry guarded by an mbarrier the publishers arrive on; up to 24 passes per team. The shared-memory arena stays at 231,424 bytes. - op.py becomes megakernel_op.py. The test covers 3 to 48 rows, rows with ties, flat rows and mixed lengths on both sides of 32 rows. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…atch 1 to 1,024 Adds the large-batch kernels as dsv4_sparse_decode/large_batch: a persistent indexer kernel (one CTA per SM, row segments in equal cost units, exact top-512 selection, ready queue) and an attention kernel launched behind it with programmatic dependent launch. One compile serves every batch size. The new op.py serves every batch size from 1 to 1,024: the megakernel up to 48 rows, the large-batch kernels above. prepare / run / precompile / error_bits take the same arguments for both; implementation_for names the one a batch size uses. Tests for the large-batch kernels and for the op (every batch size around the switch, in-place refresh, CUDA graph, argument checks). Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…erge its 17 modules into 9 files
The package is the specDSA (speculative DSA) megakernel: indexer logits, top-512 selection with
a threshold guessed from sampled logits (GVR V2), compressed-KV row load and attention in one
launch up to 48 rows, in two kernels above.
- dsv4_sparse_decode -> spec_dsa; the Dsv4SparseDecode* classes -> SpecDsa*.
- megakernel: megakernel.py (kernel, row descriptors, selection entry points),
megakernel_attention.py (attention engine and roles), megakernel_lib.py (PTX helpers,
selection primitives, L2 policy), megakernel_op.py (host side).
- large batch: large_batch.py (attention helpers, attention tail, indexer kernel),
large_batch_lib.py (selection routines, L2 policy), large_batch_op.py (host side).
- op.py stays the interface; the tests are test_cute_dsl_spec_dsa_{megakernel,large_batch,op}.py.
- Every line of code is kept. Only intra-package imports, re-export aliases and duplicate
definitions are removed, and the few names that two merged modules define differently get a
suffix. The PTX and IR of the megakernel (13 batch sizes) and of both large-batch kernels are
identical to the previous layout.
Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…as its first spec specDSA kernels are specialized for the shape of one DSA model. This adds the layer that says which: spec_dsa/config.py defines SpecDsaConfig (indexer heads, head dim, K-cache dtype and page size, compression ratio, top-K, longest row, attention heads, KV row layout, window, sink, and the batch-size limits of the megakernel and the large-batch kernels) and DS_V4_FLASH, the one spec with kernels. - The structural constants of the kernels (top-K, heads, KV row size, window, selected rows, longest row, index page size, SM count) are defined from DS_V4_FLASH. The kernel constructors take a config and reject another kernel_key (require_built_for). - op.py, megakernel_op.py and large_batch_op.py take an optional config (default DS-V4-Flash) in is_supported, precompile, prepare and implementation_for; a state keeps the spec it was prepared for; the knob tables are keyed by kernel_key; the module constants MIN_CTX_LEN, MAX_CTX_LEN, MAX_BATCH_SIZE, MEGAKERNEL_MAX_BATCH and the like are read from the spec. A config without kernels gives is_supported -> (False, reason) and NotImplementedError from precompile, prepare and implementation_for. - README.md: what differs between models, where it goes, how a model is added. - test_cute_dsl_spec_dsa_config.py: 25 tests that need no GPU; the three other test files take their shapes from the spec. - The PTX and IR of the megakernel (13 batch sizes) and of both large-batch kernels are identical to the previous commit. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
Batch 3 and 4 ran on two shared teams of 8 CTAs in several passes. Each row now gets its own team of 4 CTAs (two row groups x two V column halves of 256, one KV gather per tile), so 3 and 4 rows run in a single pass. - megakernel.py, megakernel_attention.py: knob team4 (team size, N = 256 PV MMA with the O accumulator at columns 256-511, the partial O of the partner row group handed over as BF16 through DSMEM: _merge_22). Off for every other batch size. - megakernel_op.py: knob set _BS4T (team4, 4 teams, no dry merge, no candidate-row L2 prefetch) for 3 and 4 rows; a note that compute-sanitizer synccheck only completes for 1 to 4 rows, because from 5 rows on every pass after the first re-initializes its mbarriers without invalidating them. - tests: two more 3 and 4-row cases in the megakernel test, 4 rows in the op test. The PTX and IR of the megakernel at 1, 2, 5, 8, 9, 16, 17, 32, 33 and 40 rows and of both large-batch kernels are identical to the previous commit. 3 and 4 rows: 1.229x and 1.231x -> 1.333x and 1.270x of the production chain (nsys, CUDA graph, clean L2, 4 real-capture layers, two processes). Real-capture gates 252/252 at 3 and 4 rows, synccheck clean. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…below 80 rows The megakernel is built for up to 48 rows, but at 48 rows the large-batch kernels are faster once their CTA schedule matches the rows. - config.py: DS-V4-Flash megakernel_max_batch 48 -> 47; the knob table, README and docstrings follow. - large_batch_op.py: below 80 rows the CTA schedule cuts each row into the fewest equal pieces that fit one per CTA (rows shorter than 128 units stay whole) when that gives fewer selection pieces than the even split and costs the busiest CTA at most 3% more stream work. 77 us of host time per call instead of 18 us; unchanged from 80 rows on. - tests: an independent reference of the aligned schedule, the 47 and 48-row boundary in the config, op and megakernel tests. The PTX and IR of the large-batch kernels are unchanged. Real-capture gates 252/252 at 48, 49, 56, 60, 64 and 72 rows. 48 and 72 rows: 1.047x and 1.076x -> 1.082x and 1.105x of the production chain (nsys, CUDA graph, clean L2, 4 real-capture layers, two or more processes); 48 to 79 rows 0.4% geometric mean; in a paired run 56 and 60 rows were about 2% slower. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
Adds the MXFP4 indexer K cache of DeepSeek-V4-Flash to specDSA as a second spec, DS-V4-Flash-FP4. - FP4 logits front end in the megakernel's stream CTAs and in the large-batch indexer kernel: E2M1 K pages and queries with a UE8M0 scale per 32 elements, block-scaled MXF4 MMA, the K and query scale factors copied to tensor memory with UTCCP; the epilogue is the FP8 one without the K scale, so the logits equal the production FP4 K1 kernel. - op.run takes the query scales as q_scale. - Megakernel for 1 to 22 rows, large-batch kernels from 23 rows. - The code generated for DS-V4-Flash is unchanged (PTX and IR compared). Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
… teams, large-batch merge and schedule Tail attention (tail_attention.py, tail_op.py): behind the production FP4 indexer logits kernel and the production top-K kernel, one kernel replaces the local-to-global mapping, the sparse MLA and its reduction kernel of the unfused chain. A cluster of 5 CTAs serves a row (up to 26 rows, one wave at one CTA per SM), every CTA runs the attention of its KV tiles with a local softmax maximum and the partial outputs are combined through distributed shared memory. FP4 megakernel: four shared selection + attention teams of 4 CTAs instead of two of 8, split ranges rotated over the stream CTAs per row (one stream CTA holds ~35 % of a row's candidates on real captures), two dedicated threshold-server CTAs, the sampling ladder in the row descriptor. FP4 megakernel_max_batch is 20. FP4 large batch: the last piece of a split row merges its row in one selection, the schedule verdict issues its loads before its first branch, a host CTA schedule charges a CTA for every row it touches, the selector warps finalize the last segment without waiting for the row warps, faster selection and compaction loops, and a second compile of the indexer kernel from 60 rows with six K stages and a longer selector sleep. The FP8 spec generates the same PTX / IR as before. Tests: tail attention, host schedule and knob tables (CPU), FP4 megakernel sizes 1 to 20, negative-logit regression for a signed Uint32 compaction bound, config and op switch sizes. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…ow-boundary knobs for 1 to 4 rows The FP4 stream's sample-window index pos // wst becomes umulhi(pos, magic) (exact for pos < 2^18, 8 <= wst < 2^14; checked exhaustively on the host), which shortens the math warps' per-element path: the megakernel is 1 to 4 % faster at 1 to 20 rows. The FP4 sets of 1 to 4 rows add the existing start / row-boundary knobs (descriptor prefetch at entry and one row ahead, rotated split ranges). README: the fastest path per batch size against the production FP4 chain. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…4 rows The threshold lead of the FP4 megakernel started polling the sample board only after ~4.4 us because of two start fences under the start burst: the release fence now overlaps the first poll loads (lpoll, lfence at one row). The bitwise rank search becomes a 256-bin histogram with two barriers (lbin, bit-identical threshold line), the stream CTAs read the threshold line in one round trip with tagged validation (tpoll1), the classification of buffered tiles issues four tiles' loads first (cls4) and the followers' dry pass warms the threshold store block instead of racing the lead (fwarm). All knobs are default off and set only for the FP4 sets of 1 to 4 rows; the FP8 kernels and the FP4 kernels at 5 or more rows generate the same code as before. Tests: the FP4 knob-table assertion and a test that the published threshold line does not depend on the new knobs (1 to 4 rows). Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
1 task done
… rows From 512 rows on the 1000 W board power cap lowers the SM clock from 1.63 to 1.38 GHz. The per-tile chain of the large-batch indexer kernel's math warps (about 292 SM cycles per 128 positions) is longer than the HBM time of a tile (184 ns) below 1.59 GHz, so the kernel becomes SM bound while the production indexer kernel stays HBM bound. Four knobs take instructions and hand-offs off that chain: lim_loop and lim_loop2 (the loop control is one compare against a per-segment limit), nolock (the UMMA warps meet only at a row change) and floor_mask (the floor is read less often). They are default off and set by a three-tier table: from 60 rows (sel_idle_ns 1536, kv_stages 6, lim_loop, lim_loop2, nolock), from 192 rows (plus floor_mask 30) and from 768 rows (sel_idle_ns 3072, kv_stages 8). The FP8 kernels and the FP4 kernels below 60 rows generate the same code as before. nsys against the FP4 chain, 4 passes: +3.4 % at 1024 rows, +2.9 % at 768, +1.6 % at 512, +0.3 to +1.1 % at 64 to 384 rows; 32 to 56 rows within +-0.1 %. Tests: the knob tables of the large-batch kernel and a test that the limit loop visits the same tiles as the shipped loop condition. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…-batch size, wide output stores The first dispatch tier of the indexer kernel (sel_idle_ns 1536, kv_stages 6, lim_loop, lim_loop2, nolock) now starts at the first large-batch size instead of at 60 rows: the base knobs (8 K stages, sel_idle_ns 128) are 0.9 to 2 % slower at 27 to 59 rows, and 8 stages with the loop-control trims are 3 to 4 % slower at 47 to 64 rows. The output epilogue of the attention kernel writes 32-byte stores (st.global.v8, half the store instructions of 16-byte ones; the epilogue is about 3 us of the ~13 us row latency that ends every call); an output that is only 16-byte aligned runs a second, narrower compile (every torch allocation is 32-byte aligned). nsys against the FP4 chain, 3 passes, against the previous commit: +1.0 to +2.2 % at 32 to 56 rows, within +-0.2 % at 64 to 384 rows, +0.5 to +0.6 % at 512 to 1024 rows. Tests: the large-batch knob tables and the output alignment (8 cases for 16- and 32-byte aligned outputs). Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…8 rows One row: the followers' dry pass of the threshold code (fwarm) is off; it measures 0.1 to 0.2 us slower in the nsys A / B of the unstamped kernel. Two and three rows: the candidate publish of the last row goes out in batches (b1l, b1l_u 3; at two rows also b1l_pub, b1l_upub 3). Five to eight rows: the threshold lead ranks the sampled logits with the 256-bin histogram (lbin, the published threshold line is bit for bit the same) and, at eight rows, the candidate publish of the last row goes out four per lane. Table changes only: no kernel code changes, the FP8 sets are untouched. nsys fused time, 4 chunks against 2 to 3 chunks: 1 row -0.12 us, 2 rows -0.07, 3 rows -0.27, 5 rows -0.90, 6 rows -0.54, 7 rows -0.76, 8 rows -1.28; 4 rows is unchanged. Tests: the FP4 knob-table assertion and the published-threshold-line test extended to 5 to 8 rows. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
Follow-up to #22223 (this branch is stacked on it: review the last four commits only; it is rebased onto main when #22223 merges). Adds the MXFP4 indexer K cache of DeepSeek-V4-Flash, the format the model uses, to specDSA as a second spec,
DS-V4-Flash-FP4, and serves each batch size with the implementation that is fastest against the production FP4 chain:tail_op.run_attention) that replaces the chain's local-to-global mapping, sparse MLA and reduction kernelop.runis self-contained: megakernel up to 20 rows (megakernel_max_batchof the FP4 spec) and large-batch kernels above. The tail attention is for callers that run the indexer logits and top-K kernels themselves, as the model does (tail_op.run_hybrid).tcgen05.mma. The K scale factors are transposed in shared memory to the layout UTCCP expects and copied to tensor memory per K tile; the query scales are copied per row. The epilogue is the FP8 one without the K scale, so the logits equal the production FP4 logits kernel bit for bit and the top-512 is exact.op.runtakes the query scales asq_scale.Performance, B200, clean L2 (1 GiB write + 512 MiB read before every replay), nsys kernel time of a CUDA-graph replay, mean of 4 layers of a 1M-token capture x 30 replays x 3 passes. Baseline: the production FP4 chain (FP4 paged-MQA logits, GVR V2 top-K, local-to-global, sparse MLA;
TRTLLM_ENABLE_PDL=1). The FP4 data are the real FP8 captures requantized to MXFP4 with the productionfused_cat_fp4. For 9 to 26 rows the time is the whole graph: production indexer logits, production top-K and the tail attention.Over the 40 batch sizes below the geometric mean of the speedup is 1.161 and the minimum 1.047 (at 32 rows); no size is slower than the chain. By range: 1 to 8 rows 1.300 (min 1.114), 9 to 26 rows 1.115 (min 1.074), above 26 rows 1.143 (min 1.047). The chain itself has a slow route at 13 and 14 rows (its sparse MLA takes 18 instead of 10 us), which is why the tail attention reads 1.2x there.
The K stream is HBM-bound in every path (the production indexer logits kernel runs at ~6.0 TB/s, the large-batch indexer kernel at ~6.6 TB/s), so above ~100 rows the gain is the part of the selection and attention that the large-batch kernels hide behind the stream (1.16 to 1.19x), and below 27 rows it is the shorter tail after the K stream (the chain's top-K, mapping and attention kernels take 25 to 35 us; the tail attention replaces the last two of them).
All measured batch sizes (mean over replays; p10 / p90 of the specDSA replays)
Test Coverage
Unit tests (B200),
tests/unittest/_torch/attention/sparse/deepseek_v4/:test_cute_dsl_spec_dsa_tail.py(new): 155 passed; tail attention against the FP32 reference at the cluster boundaries, short windows, -1 padded and empty selections, CUDA-graph replay, error cases, and a power-of-two mode that checks the cross-CTA combine at 2e-5.test_cute_dsl_spec_dsa_host.py(new): 134 passed (CPU only): host CTA schedule invariants, the FP8 schedule equal to the previous one on 2,324 (batch, length) mixes, knob tables.test_cute_dsl_spec_dsa_large_batch.py: 48 passed (FP8 and FP4 fixtures, mixed lengths, ties, all-equal rows, state reuse, CUDA-graph replay, negative logits: a regression test for a signed min/max of a Uint32 compaction bound).test_cute_dsl_spec_dsa_megakernel.py: 72 passed (FP4 cases for every size from 1 to 20).test_cute_dsl_spec_dsa_op.py: 91 passed;test_cute_dsl_spec_dsa_config.py: 27 passed.Also run on this branch (head 2311c91; the last commit changes the FP4 megakernel only, the large-batch and tail-attention runs below are from the previous commit, whose code is unchanged): the 252 real-capture decode steps (2 captures x 21 layers x 3 steps, 2 repeats; exact top-512 set against the chain's logits, attention against the FP32 reference) at 1, 2, 3, 4, 5, 6, 7, 8, 12, 16, 20, 21, 64 rows through the op and at 8, 16, 24, 26 rows through the tail attention: 4,536 of 4,536 runs pass (max calc_diff 6.51e-5); a random mixed-length stress with planted ties, all-equal rows and all-negative logits: 890 iterations (1,780 launches), 0 failures;
compute-sanitizer --tool synccheckon the megakernel at 1 row and on the tail kernel at 1 and 2 rows: 0 errors (runs at 2 or more rows of the megakernel did not finish under the tool).PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.