Repository navigation
[None][feat] specDSA: exact multi-CTA top-K for the tail attention hybrid - #22725
Draft
longcheng-nv wants to merge 19 commits into
Draft
longcheng-nv wants to merge 19 commits into
longcheng-nv wants to merge 19 commits into
Conversation
Add tensorrt_llm/_torch/attention/kernels/blackwell/dsv4_sparse_decode, a CuTe DSL kernel that runs the DeepSeek-V4-Flash CSA decode step (FP8 indexer logits, exact top-512, KV index mapping and sparse MLA with the attention sink) as one cooperative launch over the 148 SMs of a B200 for 1, 2, 4 or 8 rows, with its unit test. The kernel is not wired into the model yet. Made-with: Claude Code (Opus 5.5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…stage release The stream math warps read each K stage's per-token scales, and at batch > 1 the row's weights from the Q / W stage, and then release the stage with an mbarrier arrive. The arrive does not order the warp's outstanding shared-memory loads before the TMA refill (async proxy), so a delayed load could read the next tile's data. Add fence.proxy.async.shared::cta before each of these releases (5 sites). - Without the fences no failure was seen in 668,002 checked launches. With extra co-resident load / store traffic (stress-only builds), the K-stage race changed the top-512 in up to 99 % of launches. - With the fences, 0 of 378,000 mixed-length stress launches failed, 252,000 of them under that traffic. The gates (252 real cases per batch size) and the unit test pass. - Cost (nsys, clean L2): +0.31 / +0.02 / +0.02 / +0.30 us at batch 1 / 2 / 4 / 8. Made-with: Claude Code (Opus 5.5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…, batched candidate publish at batch 8 - Batch > 1: the stream CTAs' TMA warps take row 0's descriptor from the kernel-entry load and prefetch the tensor maps at entry, both before the CTA barrier; at batch 2 the team CTA also loads the row and window lengths before its barrier setup. - Batch 8: the candidate publish issues its histogram atomics in batches of two per thread, and the last row issues its first batch before the count barrier. This relies on the previous commit's fences. Fused-kernel time against the previous commit (nsys, clean L2): -0.12 / -0.04 / -1.13 us at batch 2 / 4 / 8 (a second GPU: -0.27 / -1.18 us at batch 2 / 8); batch 1 is unchanged. The gates (252 real cases per batch size) and the unit test pass, and 0 of 136,500 stress launches (mixed-length and hot-row cases) failed. Made-with: Claude Code (Opus 5.5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…1 to 48 The megakernel served 1, 2, 4 and 8 rows. It now serves every batch size from 1 to 48. - 3, 5, 6 and 7 rows use the 4- and 8-row configurations. - 9 to 16 rows: the logits of a row live in a ring of 8 shared-memory slots. A slot is reused after its row's selection reached K_DONE, which the T poller (warp 12) or the publishers observe before the row that needs the slot. - 17 to 32 rows: a 32-entry per-row service state past the stream layout, slot-free barrier phases (r >> 3) - 1 & 1, up to 16 passes per team. - 33 to 48 rows: one generation word and five state words per row, and the compressed-KV block-table slices of the stream CTAs as a ring of 32 entries, each reusable entry guarded by an mbarrier the publishers arrive on; up to 24 passes per team. The shared-memory arena stays at 231,424 bytes. - op.py becomes megakernel_op.py. The test covers 3 to 48 rows, rows with ties, flat rows and mixed lengths on both sides of 32 rows. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…atch 1 to 1,024 Adds the large-batch kernels as dsv4_sparse_decode/large_batch: a persistent indexer kernel (one CTA per SM, row segments in equal cost units, exact top-512 selection, ready queue) and an attention kernel launched behind it with programmatic dependent launch. One compile serves every batch size. The new op.py serves every batch size from 1 to 1,024: the megakernel up to 48 rows, the large-batch kernels above. prepare / run / precompile / error_bits take the same arguments for both; implementation_for names the one a batch size uses. Tests for the large-batch kernels and for the op (every batch size around the switch, in-place refresh, CUDA graph, argument checks). Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…erge its 17 modules into 9 files
The package is the specDSA (speculative DSA) megakernel: indexer logits, top-512 selection with
a threshold guessed from sampled logits (GVR V2), compressed-KV row load and attention in one
launch up to 48 rows, in two kernels above.
- dsv4_sparse_decode -> spec_dsa; the Dsv4SparseDecode* classes -> SpecDsa*.
- megakernel: megakernel.py (kernel, row descriptors, selection entry points),
megakernel_attention.py (attention engine and roles), megakernel_lib.py (PTX helpers,
selection primitives, L2 policy), megakernel_op.py (host side).
- large batch: large_batch.py (attention helpers, attention tail, indexer kernel),
large_batch_lib.py (selection routines, L2 policy), large_batch_op.py (host side).
- op.py stays the interface; the tests are test_cute_dsl_spec_dsa_{megakernel,large_batch,op}.py.
- Every line of code is kept. Only intra-package imports, re-export aliases and duplicate
definitions are removed, and the few names that two merged modules define differently get a
suffix. The PTX and IR of the megakernel (13 batch sizes) and of both large-batch kernels are
identical to the previous layout.
Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…as its first spec specDSA kernels are specialized for the shape of one DSA model. This adds the layer that says which: spec_dsa/config.py defines SpecDsaConfig (indexer heads, head dim, K-cache dtype and page size, compression ratio, top-K, longest row, attention heads, KV row layout, window, sink, and the batch-size limits of the megakernel and the large-batch kernels) and DS_V4_FLASH, the one spec with kernels. - The structural constants of the kernels (top-K, heads, KV row size, window, selected rows, longest row, index page size, SM count) are defined from DS_V4_FLASH. The kernel constructors take a config and reject another kernel_key (require_built_for). - op.py, megakernel_op.py and large_batch_op.py take an optional config (default DS-V4-Flash) in is_supported, precompile, prepare and implementation_for; a state keeps the spec it was prepared for; the knob tables are keyed by kernel_key; the module constants MIN_CTX_LEN, MAX_CTX_LEN, MAX_BATCH_SIZE, MEGAKERNEL_MAX_BATCH and the like are read from the spec. A config without kernels gives is_supported -> (False, reason) and NotImplementedError from precompile, prepare and implementation_for. - README.md: what differs between models, where it goes, how a model is added. - test_cute_dsl_spec_dsa_config.py: 25 tests that need no GPU; the three other test files take their shapes from the spec. - The PTX and IR of the megakernel (13 batch sizes) and of both large-batch kernels are identical to the previous commit. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
Batch 3 and 4 ran on two shared teams of 8 CTAs in several passes. Each row now gets its own team of 4 CTAs (two row groups x two V column halves of 256, one KV gather per tile), so 3 and 4 rows run in a single pass. - megakernel.py, megakernel_attention.py: knob team4 (team size, N = 256 PV MMA with the O accumulator at columns 256-511, the partial O of the partner row group handed over as BF16 through DSMEM: _merge_22). Off for every other batch size. - megakernel_op.py: knob set _BS4T (team4, 4 teams, no dry merge, no candidate-row L2 prefetch) for 3 and 4 rows; a note that compute-sanitizer synccheck only completes for 1 to 4 rows, because from 5 rows on every pass after the first re-initializes its mbarriers without invalidating them. - tests: two more 3 and 4-row cases in the megakernel test, 4 rows in the op test. The PTX and IR of the megakernel at 1, 2, 5, 8, 9, 16, 17, 32, 33 and 40 rows and of both large-batch kernels are identical to the previous commit. 3 and 4 rows: 1.229x and 1.231x -> 1.333x and 1.270x of the production chain (nsys, CUDA graph, clean L2, 4 real-capture layers, two processes). Real-capture gates 252/252 at 3 and 4 rows, synccheck clean. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…below 80 rows The megakernel is built for up to 48 rows, but at 48 rows the large-batch kernels are faster once their CTA schedule matches the rows. - config.py: DS-V4-Flash megakernel_max_batch 48 -> 47; the knob table, README and docstrings follow. - large_batch_op.py: below 80 rows the CTA schedule cuts each row into the fewest equal pieces that fit one per CTA (rows shorter than 128 units stay whole) when that gives fewer selection pieces than the even split and costs the busiest CTA at most 3% more stream work. 77 us of host time per call instead of 18 us; unchanged from 80 rows on. - tests: an independent reference of the aligned schedule, the 47 and 48-row boundary in the config, op and megakernel tests. The PTX and IR of the large-batch kernels are unchanged. Real-capture gates 252/252 at 48, 49, 56, 60, 64 and 72 rows. 48 and 72 rows: 1.047x and 1.076x -> 1.082x and 1.105x of the production chain (nsys, CUDA graph, clean L2, 4 real-capture layers, two or more processes); 48 to 79 rows 0.4% geometric mean; in a paired run 56 and 60 rows were about 2% slower. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
Adds the MXFP4 indexer K cache of DeepSeek-V4-Flash to specDSA as a second spec, DS-V4-Flash-FP4. - FP4 logits front end in the megakernel's stream CTAs and in the large-batch indexer kernel: E2M1 K pages and queries with a UE8M0 scale per 32 elements, block-scaled MXF4 MMA, the K and query scale factors copied to tensor memory with UTCCP; the epilogue is the FP8 one without the K scale, so the logits equal the production FP4 K1 kernel. - op.run takes the query scales as q_scale. - Megakernel for 1 to 22 rows, large-batch kernels from 23 rows. - The code generated for DS-V4-Flash is unchanged (PTX and IR compared). Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
… teams, large-batch merge and schedule Tail attention (tail_attention.py, tail_op.py): behind the production FP4 indexer logits kernel and the production top-K kernel, one kernel replaces the local-to-global mapping, the sparse MLA and its reduction kernel of the unfused chain. A cluster of 5 CTAs serves a row (up to 26 rows, one wave at one CTA per SM), every CTA runs the attention of its KV tiles with a local softmax maximum and the partial outputs are combined through distributed shared memory. FP4 megakernel: four shared selection + attention teams of 4 CTAs instead of two of 8, split ranges rotated over the stream CTAs per row (one stream CTA holds ~35 % of a row's candidates on real captures), two dedicated threshold-server CTAs, the sampling ladder in the row descriptor. FP4 megakernel_max_batch is 20. FP4 large batch: the last piece of a split row merges its row in one selection, the schedule verdict issues its loads before its first branch, a host CTA schedule charges a CTA for every row it touches, the selector warps finalize the last segment without waiting for the row warps, faster selection and compaction loops, and a second compile of the indexer kernel from 60 rows with six K stages and a longer selector sleep. The FP8 spec generates the same PTX / IR as before. Tests: tail attention, host schedule and knob tables (CPU), FP4 megakernel sizes 1 to 20, negative-logit regression for a signed Uint32 compaction bound, config and op switch sizes. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…ow-boundary knobs for 1 to 4 rows The FP4 stream's sample-window index pos // wst becomes umulhi(pos, magic) (exact for pos < 2^18, 8 <= wst < 2^14; checked exhaustively on the host), which shortens the math warps' per-element path: the megakernel is 1 to 4 % faster at 1 to 20 rows. The FP4 sets of 1 to 4 rows add the existing start / row-boundary knobs (descriptor prefetch at entry and one row ahead, rotated split ranges). README: the fastest path per batch size against the production FP4 chain. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…4 rows The threshold lead of the FP4 megakernel started polling the sample board only after ~4.4 us because of two start fences under the start burst: the release fence now overlaps the first poll loads (lpoll, lfence at one row). The bitwise rank search becomes a 256-bin histogram with two barriers (lbin, bit-identical threshold line), the stream CTAs read the threshold line in one round trip with tagged validation (tpoll1), the classification of buffered tiles issues four tiles' loads first (cls4) and the followers' dry pass warms the threshold store block instead of racing the lead (fwarm). All knobs are default off and set only for the FP4 sets of 1 to 4 rows; the FP8 kernels and the FP4 kernels at 5 or more rows generate the same code as before. Tests: the FP4 knob-table assertion and a test that the published threshold line does not depend on the new knobs (1 to 4 rows). Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
… rows From 512 rows on the 1000 W board power cap lowers the SM clock from 1.63 to 1.38 GHz. The per-tile chain of the large-batch indexer kernel's math warps (about 292 SM cycles per 128 positions) is longer than the HBM time of a tile (184 ns) below 1.59 GHz, so the kernel becomes SM bound while the production indexer kernel stays HBM bound. Four knobs take instructions and hand-offs off that chain: lim_loop and lim_loop2 (the loop control is one compare against a per-segment limit), nolock (the UMMA warps meet only at a row change) and floor_mask (the floor is read less often). They are default off and set by a three-tier table: from 60 rows (sel_idle_ns 1536, kv_stages 6, lim_loop, lim_loop2, nolock), from 192 rows (plus floor_mask 30) and from 768 rows (sel_idle_ns 3072, kv_stages 8). The FP8 kernels and the FP4 kernels below 60 rows generate the same code as before. nsys against the FP4 chain, 4 passes: +3.4 % at 1024 rows, +2.9 % at 768, +1.6 % at 512, +0.3 to +1.1 % at 64 to 384 rows; 32 to 56 rows within +-0.1 %. Tests: the knob tables of the large-batch kernel and a test that the limit loop visits the same tiles as the shipped loop condition. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
longcheng-nv
force-pushed
the
perf/dsv4-specdsa-k2p
branch
from
November 21, 2026 05:59
0d321ca to
77bb430
Compare
…-batch size, wide output stores The first dispatch tier of the indexer kernel (sel_idle_ns 1536, kv_stages 6, lim_loop, lim_loop2, nolock) now starts at the first large-batch size instead of at 60 rows: the base knobs (8 K stages, sel_idle_ns 128) are 0.9 to 2 % slower at 27 to 59 rows, and 8 stages with the loop-control trims are 3 to 4 % slower at 47 to 64 rows. The output epilogue of the attention kernel writes 32-byte stores (st.global.v8, half the store instructions of 16-byte ones; the epilogue is about 3 us of the ~13 us row latency that ends every call); an output that is only 16-byte aligned runs a second, narrower compile (every torch allocation is 32-byte aligned). nsys against the FP4 chain, 3 passes, against the previous commit: +1.0 to +2.2 % at 32 to 56 rows, within +-0.2 % at 64 to 384 rows, +0.5 to +0.6 % at 512 to 1024 rows. Tests: the large-batch knob tables and the output alignment (8 cases for 16- and 32-byte aligned outputs). Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…8 rows One row: the followers' dry pass of the threshold code (fwarm) is off; it measures 0.1 to 0.2 us slower in the nsys A / B of the unstamped kernel. Two and three rows: the candidate publish of the last row goes out in batches (b1l, b1l_u 3; at two rows also b1l_pub, b1l_upub 3). Five to eight rows: the threshold lead ranks the sampled logits with the 256-bin histogram (lbin, the published threshold line is bit for bit the same) and, at eight rows, the candidate publish of the last row goes out four per lane. Table changes only: no kernel code changes, the FP8 sets are untouched. nsys fused time, 4 chunks against 2 to 3 chunks: 1 row -0.12 us, 2 rows -0.07, 3 rows -0.27, 5 rows -0.90, 6 rows -0.54, 7 rows -0.76, 8 rows -1.28; 4 rows is unchanged. Tests: the FP4 knob-table assertion and the published-threshold-line test extended to 5 to 8 rows. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…brid topk_multicta (topk_multicta.py, host side topk_multicta_op.py): one cluster of g CTAs per row reads each row once, a sampled threshold bins the candidates, the CTAs exchange per-bin counts through distributed shared memory and write everything above the crossing bin directly; a row that fails the count certificate takes an exact four-level radix selection over the whole row, so the result is exact for every input. tail_op.run_hybrid uses it when every row has at least 16,384 positions. The radix fallback had a write-after-read race between its levels (found under compute-sanitizer timing, invisible to the stress runs, the gates and synccheck); one barrier per level fixes it, racecheck reports no errors and a test fuzzes the kernel schedule. Tests: test_cute_dsl_spec_dsa_topk.py. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…i-CTA top-K (PDL) The multi-CTA top-K executes griddepcontrol.launch_dependents after its stream phase and the tail attention is launched with the programmatic attribute. Its clusters start on the SMs the top-K leaves free and run their barrier and TMEM setup, the query load and the window tile while the top-K ranks the crossing bin; the griddepcontrol.wait in front of the first read of the selection was already there. run_hybrid(pdl=None) turns both on with the multi-CTA top-K (pdl=False keeps the plain stream order); both programmatic variants are compiled by precompile, which must run before a CUDA-graph capture. The PTX of the default-config kernels is unchanged, the trigger variant has one added instruction. nsys against the same hybrid in plain launch order, 3 passes, 9 to 24 rows: -0.53 us on average (-0.83 %), up to -1.4 us at 10 rows and ~0 at 24 rows (the top-K holds 120 SMs there). The trigger point matters: at the kernel start +3.9 us at 9 rows, after the stream phase -0.83 %. Tests: an ordering test with a slow stand-in top-K that triggers early (a tail attention without griddepcontrol.wait fails it) and the validation / API tests of the programmatic variants (12 new tests in each of the tail-attention and top-K test files). Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…ttention, 8-CTA top-K clusters at 13 to 15 rows, PDL up to 24 rows The CTAs of the tail attention compute and send the 128-column output slices in the order of their rank (CTA r starts with its own slice and then serves the owners r + 1, r + 2, ... one after the other), so that every owner receives from one CTA at a time; before, all senders queued at owner 0, then at owner 1, and so on, and the distributed shared memory of an SM takes about 28 bytes per clock. The slices are sent before a CTA waits for the (max, sum) pairs of its peers; the results are bit-identical. With the programmatic launch the CTAs of the selected tiles bring their share of the block-table row into L2 while the top-K kernel finishes. The multi-CTA top-K uses clusters of 8 CTAs at 13 to 15 rows (6 before; 0.5 to 1.2 us faster behind the programmatic launch). run_hybrid launches programmatically by default up to 24 rows (tail_op.PDL_MAX_BATCH): at 25 and 26 rows the 26 clusters of one wave are all but full and the early launch costs up to 3 us. The README dispatch becomes: megakernel up to 4 rows, tail attention from 5 to 26 rows, large batch above. nsys against the FP4 chain, 3 passes, fused time of the hybrid against the previous commit: -0.5 to -2.4 us at 5 to 24 rows (mean -1.2 us at 9 to 24 rows). From 5 rows the hybrid is 1 to 3 us faster than the megakernel; at 3 and 4 rows the megakernel stays 2.8 to 3.4 us ahead. Tests: the default launch mode and the stage order with a slow trigger kernel at 3, 24, 25 and 26 rows. Made-with: Claude Code (Sonnet 5.5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
longcheng-nv
force-pushed
the
perf/dsv4-specdsa-k2p
branch
from
November 21, 2026 09:52
77bb430 to
7f4c603
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
Follow-up to #22678 (stacked on it: review the last commit only; it is rebased onto main when #22678 merges). The tail attention of #22678 runs behind the production FP4 indexer logits kernel and the production top-K kernel (GVR V2). This adds an exact multi-CTA top-512 kernel for the same slot,
topk_multicta(topk_multicta.py, host sidetopk_multicta_op.py), andtail_op.run_hybriduses it when every row has at least 16,384 compressed positions (topk_multicta_op.is_profitable).synccheck/memcheckdid not show: it appeared undercompute-sanitizertiming as 128 to 317 of the 512 output slots left unwritten, and rows of at most 8,192 positions always take that path. One barrier per level fixes it.racecheckreports 2 errors on the kernel without the barrier and 0 with it; a regression test rewrites the kernel source with nanosleeps at the barriers (a schedule fuzz) and fails deterministically without the barrier (47 % of the rows of a fuzzed batch), and passes with it (0 of 24,000).topk_multicta_op.run(logits, topk, ctx_lens, *, config=None),precompile,is_supported(device, config),is_batch_size_supported,cluster_size; one compile per (device, cluster size), 1 to 26 rows, every argument validated.tail_op.run_hybridchange; the generated code of the FP8 and FP4 kernels of [None][feat] specDSA: MXFP4 indexer K cache (DS-V4-Flash-FP4) and tail attention for 9 to 26 rows #22678 is unchanged.Performance, B200, clean L2 (1 GiB write + 512 MiB read before every replay), nsys kernel time of a CUDA-graph replay, mean of 4 layers of a 1M-token capture x 30 replays x 3 passes. Baseline: the production FP4 chain. The same dispatch as #22678: megakernel for 1 to 8 rows, the tail attention for 9 to 26 rows (now with the multi-CTA top-K), large batch above. For 9 to 26 rows the time is the whole graph: production indexer logits, top-K and the tail attention. Production top-K and the multi-CTA top-K run in one process per chunk with identical batch-size lists:
The step gets 2.4 % shorter on average (1.3 % at 15 rows to 3.6 % at 18 rows); every size gains. The multi-CTA kernel takes 9.2 to 11.4 us against 9.9 to 13.2 us of the production top-K.
Over the 40 batch sizes the geometric mean of the speedup is 1.173 (#22678 alone: 1.161) and the minimum 1.047 (at 32 rows); no size is slower than the chain.
All 40 batch sizes
Test Coverage
tests/unittest/_torch/attention/sparse/deepseek_v4/test_cute_dsl_spec_dsa_topk.py(new): 180 passed; exact set againsttorch.topkon 15 row kinds (Gaussian, heavy tail, quantized, all equal, +-0, subnormals, signed and negative-only rows, planted ties) at the batch-size boundaries and for both specs, 26 edge lengths, mixed lengths, forced fallback, garbage beyond the row length, other buffer sizes, output canaries, misaligned logits, CUDA-graph replay,precompile/ capture combinations, one compile per cluster size, argument refusals,run_hybridagainst the FP64 attention reference, and the warp-skew regression test of the fallback. A mutation check (5 kernel mutants: dropped tie cap, wrong rank, leaked candidates, short-row path, above-boundary count) kills all 5.test_cute_dsl_spec_dsa_tail.py: 155 passed on this branch (unchanged).Also run: the 252 real-capture decode steps (2 captures x 21 layers x 3 steps, 2 repeats; exact top-512 set against the chain's logits, attention against the FP32 reference) with the multi-CTA top-K in place of the production one, through the chain (production indexer logits -> top-K -> mapping + attention) and through
run_hybrid, at 8, 12, 16, 24 and 26 rows and with CUDA-graph replay: 3,276 of 3,276 runs pass (max calc_diff 4.29e-5); a stress of 112,011 row checks (production and forced-fallback kernels on the same rows, 15 kinds, lengths 1 to 2^18, all four cluster sizes), 0 failing;compute-sanitizersynccheck and memcheck at 1 and 2 rows and racecheck at 1, 13, 17 and 25 rows: 0 errors.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.