Skip to content

[None][feat] specDSA: exact multi-CTA top-K for the tail attention hybrid - #22725

Draft
longcheng-nv wants to merge 19 commits into
NVIDIA:mainfrom
longcheng-nv:perf/dsv4-specdsa-k2p
Draft

longcheng-nv wants to merge 19 commits into
NVIDIA:mainfrom
longcheng-nv:perf/dsv4-specdsa-k2p

Conversation

@longcheng-nv

Copy link
Copy Markdown
Collaborator

@coderabbitai summary

Description

Follow-up to #22678 (stacked on it: review the last commit only; it is rebased onto main when #22678 merges). The tail attention of #22678 runs behind the production FP4 indexer logits kernel and the production top-K kernel (GVR V2). This adds an exact multi-CTA top-512 kernel for the same slot, topk_multicta (topk_multicta.py, host side topk_multicta_op.py), and tail_op.run_hybrid uses it when every row has at least 16,384 compressed positions (topk_multicta_op.is_profitable).

  • One cluster of g CTAs per row (8 CTAs for 1 to 12 rows, 6 for 13 to 16, 5 for 17 to 24, 4 for 25 and 26) reads each row once. A threshold sampled from the row bins the candidates, the CTAs exchange their per-bin counts through distributed shared memory, everything above the crossing bin is written directly and the lead CTA ranks the members of the crossing bin.
  • The count certificate (at least 512 candidates, bounded crossing bin) is checked in the kernel; a row that fails it takes an exact four-level radix selection over the whole row, so the result is exact for every input. A fallback row costs about 6 us more; none of the 630 real rows of the captures takes it. The value multiset of the 512 is exact; which of several equal values at the 512th rank are taken is unspecified in the fallback (the fast path takes the smaller position). NaN logits are outside the contract.
  • The radix fallback had a write-after-read race between its levels (threads re-initialised the per-source counters while slower warps still read them) that the stress runs, the real-capture gates and synccheck / memcheck did not show: it appeared under compute-sanitizer timing as 128 to 317 of the 512 output slots left unwritten, and rows of at most 8,192 positions always take that path. One barrier per level fixes it. racecheck reports 2 errors on the kernel without the barrier and 0 with it; a regression test rewrites the kernel source with nanosleeps at the barriers (a schedule fuzz) and fails deterministically without the barrier (47 % of the rows of a fuzzed batch), and passes with it (0 of 24,000).
  • API: topk_multicta_op.run(logits, topk, ctx_lens, *, config=None), precompile, is_supported(device, config), is_batch_size_supported, cluster_size; one compile per (device, cluster size), 1 to 26 rows, every argument validated.
  • Only new files and tail_op.run_hybrid change; the generated code of the FP8 and FP4 kernels of [None][feat] specDSA: MXFP4 indexer K cache (DS-V4-Flash-FP4) and tail attention for 9 to 26 rows #22678 is unchanged.

Performance, B200, clean L2 (1 GiB write + 512 MiB read before every replay), nsys kernel time of a CUDA-graph replay, mean of 4 layers of a 1M-token capture x 30 replays x 3 passes. Baseline: the production FP4 chain. The same dispatch as #22678: megakernel for 1 to 8 rows, the tail attention for 9 to 26 rows (now with the multi-CTA top-K), large batch above. For 9 to 26 rows the time is the whole graph: production indexer logits, top-K and the tail attention. Production top-K and the multi-CTA top-K run in one process per chunk with identical batch-size lists:

rows speedup, production top-K speedup, multi-CTA top-K gain step with production top-K (us) step with multi-CTA top-K (us)
9 1.133 1.150 +1.5 % 49.2 48.4
10 1.116 1.152 +3.2 % 54.1 52.5
11 1.128 1.151 +2.0 % 57.1 56.0
12 1.136 1.163 +2.4 % 60.0 58.6
13 1.236 1.259 +1.9 % 63.5 62.3
14 1.218 1.241 +1.9 % 67.1 65.9
15 1.109 1.123 +1.3 % 69.4 68.5
16 1.109 1.136 +2.5 % 73.1 71.4
17 1.098 1.130 +2.9 % 77.4 75.2
18 1.089 1.128 +3.6 % 79.2 76.5
19 1.087 1.123 +3.3 % 83.1 80.5
20 1.089 1.117 +2.6 % 86.9 84.7
21 1.095 1.124 +2.6 % 88.8 86.5
22 1.086 1.120 +3.1 % 93.0 90.2
23 1.077 1.108 +2.9 % 95.0 92.4
24 1.077 1.100 +2.1 % 97.3 95.3
25 1.077 1.102 +2.3 % 101.6 99.3
26 1.076 1.096 +1.9 % 104.9 102.9

The step gets 2.4 % shorter on average (1.3 % at 15 rows to 3.6 % at 18 rows); every size gains. The multi-CTA kernel takes 9.2 to 11.4 us against 9.9 to 13.2 us of the production top-K.

rows chain (us) specDSA (us) speedup implementation
1 29.4 18.0 1.629 megakernel
2 33.5 21.1 1.592 megakernel
4 40.5 30.3 1.337 megakernel
8 53.9 47.9 1.126 megakernel
9 55.7 48.4 1.150 tail attention + multi-CTA top-K
12 68.1 58.6 1.163 tail attention + multi-CTA top-K
16 81.1 71.4 1.136 tail attention + multi-CTA top-K
20 94.6 84.7 1.117 tail attention + multi-CTA top-K
24 104.8 95.3 1.100 tail attention + multi-CTA top-K
32 131.2 125.3 1.047 large batch
40 161.2 144.1 1.119 large batch
48 184.8 164.3 1.125 large batch
64 231.3 205.2 1.127 large batch
100 346.8 298.0 1.164 large batch
148 498.0 418.8 1.189 large batch
256 847.6 710.9 1.192 large batch
512 1670.4 1444.0 1.157 large batch
1024 3322.4 3017.7 1.101 large batch

specDSA FP4 vs the FP4 chain

Over the 40 batch sizes the geometric mean of the speedup is 1.173 (#22678 alone: 1.161) and the minimum 1.047 (at 32 rows); no size is slower than the chain.

All 40 batch sizes
rows chain (us) specDSA (us) speedup implementation
1 29.4 18.0 1.629 megakernel
2 33.5 21.1 1.592 megakernel
3 37.4 26.2 1.425 megakernel
4 40.5 30.3 1.337 megakernel
5 42.7 37.0 1.153 megakernel
6 45.9 40.2 1.144 megakernel
7 49.0 44.0 1.114 megakernel
8 53.9 47.9 1.126 megakernel
9 55.7 48.4 1.150 tail attention + multi-CTA top-K
10 60.4 52.5 1.152 tail attention + multi-CTA top-K
11 64.4 56.0 1.151 tail attention + multi-CTA top-K
12 68.1 58.6 1.163 tail attention + multi-CTA top-K
13 78.5 62.3 1.259 tail attention + multi-CTA top-K
14 81.8 65.9 1.241 tail attention + multi-CTA top-K
15 76.9 68.5 1.123 tail attention + multi-CTA top-K
16 81.1 71.4 1.136 tail attention + multi-CTA top-K
17 85.0 75.2 1.130 tail attention + multi-CTA top-K
18 86.2 76.5 1.128 tail attention + multi-CTA top-K
19 90.3 80.5 1.123 tail attention + multi-CTA top-K
20 94.6 84.7 1.117 tail attention + multi-CTA top-K
21 97.2 86.5 1.124 tail attention + multi-CTA top-K
22 101.0 90.2 1.120 tail attention + multi-CTA top-K
23 102.4 92.4 1.108 tail attention + multi-CTA top-K
24 104.8 95.3 1.100 tail attention + multi-CTA top-K
32 131.2 125.3 1.047 large batch
40 161.2 144.1 1.119 large batch
47 180.5 162.1 1.113 large batch
48 184.8 164.3 1.125 large batch
56 208.8 186.0 1.123 large batch
64 231.3 205.2 1.127 large batch
80 286.2 245.7 1.165 large batch
100 346.8 298.0 1.164 large batch
128 434.2 373.2 1.163 large batch
148 498.0 418.8 1.189 large batch
192 646.6 541.0 1.195 large batch
256 847.6 710.9 1.192 large batch
384 1257.7 1066.8 1.179 large batch
512 1670.4 1444.0 1.157 large batch
768 2520.2 2222.7 1.134 large batch
1024 3322.4 3017.7 1.101 large batch

Test Coverage

  • tests/unittest/_torch/attention/sparse/deepseek_v4/test_cute_dsl_spec_dsa_topk.py (new): 180 passed; exact set against torch.topk on 15 row kinds (Gaussian, heavy tail, quantized, all equal, +-0, subnormals, signed and negative-only rows, planted ties) at the batch-size boundaries and for both specs, 26 edge lengths, mixed lengths, forced fallback, garbage beyond the row length, other buffer sizes, output canaries, misaligned logits, CUDA-graph replay, precompile / capture combinations, one compile per cluster size, argument refusals, run_hybrid against the FP64 attention reference, and the warp-skew regression test of the fallback. A mutation check (5 kernel mutants: dropped tie cap, wrong rank, leaked candidates, short-row path, above-boundary count) kills all 5.
  • test_cute_dsl_spec_dsa_tail.py: 155 passed on this branch (unchanged).

Also run: the 252 real-capture decode steps (2 captures x 21 layers x 3 steps, 2 repeats; exact top-512 set against the chain's logits, attention against the FP32 reference) with the multi-CTA top-K in place of the production one, through the chain (production indexer logits -> top-K -> mapping + attention) and through run_hybrid, at 8, 12, 16, 24 and 26 rows and with CUDA-graph replay: 3,276 of 3,276 runs pass (max calc_diff 4.29e-5); a stress of 112,011 row checks (production and forced-fallback kernels on the same rows, 15 kinds, lengths 1 to 2^18, all four cluster sizes), 0 failing; compute-sanitizer synccheck and memcheck at 1 and 2 rows and racecheck at 1, 13, 17 and 25 rows: 0 errors.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

longcheng-nv and others added 13 commits November 19, 2026 14:28
Add tensorrt_llm/_torch/attention/kernels/blackwell/dsv4_sparse_decode, a
CuTe DSL kernel that runs the DeepSeek-V4-Flash CSA decode step (FP8 indexer
logits, exact top-512, KV index mapping and sparse MLA with the attention
sink) as one cooperative launch over the 148 SMs of a B200 for 1, 2, 4 or 8
rows, with its unit test. The kernel is not wired into the model yet.

Made-with: Claude Code (Opus 5.5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…stage release

The stream math warps read each K stage's per-token scales, and at batch > 1 the row's weights from the Q / W stage, and
then release the stage with an mbarrier arrive. The arrive does not order the warp's outstanding shared-memory loads
before the TMA refill (async proxy), so a delayed load could read the next tile's data. Add fence.proxy.async.shared::cta
before each of these releases (5 sites).

- Without the fences no failure was seen in 668,002 checked launches. With extra co-resident load / store traffic
  (stress-only builds), the K-stage race changed the top-512 in up to 99 % of launches.
- With the fences, 0 of 378,000 mixed-length stress launches failed, 252,000 of them under that traffic. The gates (252
  real cases per batch size) and the unit test pass.
- Cost (nsys, clean L2): +0.31 / +0.02 / +0.02 / +0.30 us at batch 1 / 2 / 4 / 8.

Made-with: Claude Code (Opus 5.5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…, batched candidate publish at batch 8

- Batch > 1: the stream CTAs' TMA warps take row 0's descriptor from the kernel-entry load and prefetch the tensor
  maps at entry, both before the CTA barrier; at batch 2 the team CTA also loads the row and window lengths before its
  barrier setup.
- Batch 8: the candidate publish issues its histogram atomics in batches of two per thread, and the last row issues its
  first batch before the count barrier. This relies on the previous commit's fences.

Fused-kernel time against the previous commit (nsys, clean L2): -0.12 / -0.04 / -1.13 us at batch 2 / 4 / 8
(a second GPU: -0.27 / -1.18 us at batch 2 / 8); batch 1 is unchanged. The gates (252 real cases per batch size) and
the unit test pass, and 0 of 136,500 stress launches (mixed-length and hot-row cases) failed.

Made-with: Claude Code (Opus 5.5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…1 to 48

The megakernel served 1, 2, 4 and 8 rows. It now serves every batch size from 1 to 48.

- 3, 5, 6 and 7 rows use the 4- and 8-row configurations.
- 9 to 16 rows: the logits of a row live in a ring of 8 shared-memory slots. A slot is reused after
  its row's selection reached K_DONE, which the T poller (warp 12) or the publishers observe before
  the row that needs the slot.
- 17 to 32 rows: a 32-entry per-row service state past the stream layout, slot-free barrier phases
  (r >> 3) - 1 & 1, up to 16 passes per team.
- 33 to 48 rows: one generation word and five state words per row, and the compressed-KV block-table
  slices of the stream CTAs as a ring of 32 entries, each reusable entry guarded by an mbarrier the
  publishers arrive on; up to 24 passes per team. The shared-memory arena stays at 231,424 bytes.
- op.py becomes megakernel_op.py. The test covers 3 to 48 rows, rows with ties, flat rows and mixed
  lengths on both sides of 32 rows.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…atch 1 to 1,024

Adds the large-batch kernels as dsv4_sparse_decode/large_batch: a persistent indexer kernel (one CTA
per SM, row segments in equal cost units, exact top-512 selection, ready queue) and an attention
kernel launched behind it with programmatic dependent launch. One compile serves every batch size.

The new op.py serves every batch size from 1 to 1,024: the megakernel up to 48 rows, the
large-batch kernels above. prepare / run / precompile / error_bits take the same arguments for both;
implementation_for names the one a batch size uses. Tests for the large-batch kernels and for the
op (every batch size around the switch, in-place refresh, CUDA graph, argument checks).

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…erge its 17 modules into 9 files

The package is the specDSA (speculative DSA) megakernel: indexer logits, top-512 selection with
a threshold guessed from sampled logits (GVR V2), compressed-KV row load and attention in one
launch up to 48 rows, in two kernels above.

- dsv4_sparse_decode -> spec_dsa; the Dsv4SparseDecode* classes -> SpecDsa*.
- megakernel: megakernel.py (kernel, row descriptors, selection entry points),
  megakernel_attention.py (attention engine and roles), megakernel_lib.py (PTX helpers,
  selection primitives, L2 policy), megakernel_op.py (host side).
- large batch: large_batch.py (attention helpers, attention tail, indexer kernel),
  large_batch_lib.py (selection routines, L2 policy), large_batch_op.py (host side).
- op.py stays the interface; the tests are test_cute_dsl_spec_dsa_{megakernel,large_batch,op}.py.
- Every line of code is kept. Only intra-package imports, re-export aliases and duplicate
  definitions are removed, and the few names that two merged modules define differently get a
  suffix. The PTX and IR of the megakernel (13 batch sizes) and of both large-batch kernels are
  identical to the previous layout.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…as its first spec

specDSA kernels are specialized for the shape of one DSA model. This adds the layer that says
which: spec_dsa/config.py defines SpecDsaConfig (indexer heads, head dim, K-cache dtype and page
size, compression ratio, top-K, longest row, attention heads, KV row layout, window, sink, and
the batch-size limits of the megakernel and the large-batch kernels) and DS_V4_FLASH, the one
spec with kernels.

- The structural constants of the kernels (top-K, heads, KV row size, window, selected rows,
  longest row, index page size, SM count) are defined from DS_V4_FLASH. The kernel constructors
  take a config and reject another kernel_key (require_built_for).
- op.py, megakernel_op.py and large_batch_op.py take an optional config (default DS-V4-Flash) in
  is_supported, precompile, prepare and implementation_for; a state keeps the spec it was
  prepared for; the knob tables are keyed by kernel_key; the module constants MIN_CTX_LEN,
  MAX_CTX_LEN, MAX_BATCH_SIZE, MEGAKERNEL_MAX_BATCH and the like are read from the spec. A
  config without kernels gives is_supported -> (False, reason) and NotImplementedError from
  precompile, prepare and implementation_for.
- README.md: what differs between models, where it goes, how a model is added.
- test_cute_dsl_spec_dsa_config.py: 25 tests that need no GPU; the three other test files take
  their shapes from the spec.
- The PTX and IR of the megakernel (13 batch sizes) and of both large-batch kernels are
  identical to the previous commit.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
Batch 3 and 4 ran on two shared teams of 8 CTAs in several passes. Each row now gets its
own team of 4 CTAs (two row groups x two V column halves of 256, one KV gather per tile),
so 3 and 4 rows run in a single pass.

- megakernel.py, megakernel_attention.py: knob team4 (team size, N = 256 PV MMA with the
  O accumulator at columns 256-511, the partial O of the partner row group handed over as
  BF16 through DSMEM: _merge_22). Off for every other batch size.
- megakernel_op.py: knob set _BS4T (team4, 4 teams, no dry merge, no candidate-row L2
  prefetch) for 3 and 4 rows; a note that compute-sanitizer synccheck only completes for 1
  to 4 rows, because from 5 rows on every pass after the first re-initializes its mbarriers
  without invalidating them.
- tests: two more 3 and 4-row cases in the megakernel test, 4 rows in the op test.

The PTX and IR of the megakernel at 1, 2, 5, 8, 9, 16, 17, 32, 33 and 40 rows and of both
large-batch kernels are identical to the previous commit. 3 and 4 rows: 1.229x and 1.231x
-> 1.333x and 1.270x of the production chain (nsys, CUDA graph, clean L2, 4 real-capture
layers, two processes). Real-capture gates 252/252 at 3 and 4 rows, synccheck clean.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…below 80 rows

The megakernel is built for up to 48 rows, but at 48 rows the large-batch kernels are
faster once their CTA schedule matches the rows.

- config.py: DS-V4-Flash megakernel_max_batch 48 -> 47; the knob table, README and
  docstrings follow.
- large_batch_op.py: below 80 rows the CTA schedule cuts each row into the fewest equal
  pieces that fit one per CTA (rows shorter than 128 units stay whole) when that gives fewer
  selection pieces than the even split and costs the busiest CTA at most 3% more stream
  work. 77 us of host time per call instead of 18 us; unchanged from 80 rows on.
- tests: an independent reference of the aligned schedule, the 47 and 48-row boundary in
  the config, op and megakernel tests.

The PTX and IR of the large-batch kernels are unchanged. Real-capture gates 252/252 at 48,
49, 56, 60, 64 and 72 rows. 48 and 72 rows: 1.047x and 1.076x -> 1.082x and 1.105x of the
production chain (nsys, CUDA graph, clean L2, 4 real-capture layers, two or more
processes); 48 to 79 rows 0.4% geometric mean; in a paired run 56 and 60 rows were about 2%
slower.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
Adds the MXFP4 indexer K cache of DeepSeek-V4-Flash to specDSA as a second
spec, DS-V4-Flash-FP4.

- FP4 logits front end in the megakernel's stream CTAs and in the large-batch
  indexer kernel: E2M1 K pages and queries with a UE8M0 scale per 32 elements,
  block-scaled MXF4 MMA, the K and query scale factors copied to tensor memory
  with UTCCP; the epilogue is the FP8 one without the K scale, so the logits
  equal the production FP4 K1 kernel.
- op.run takes the query scales as q_scale.
- Megakernel for 1 to 22 rows, large-batch kernels from 23 rows.
- The code generated for DS-V4-Flash is unchanged (PTX and IR compared).

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
… teams, large-batch merge and schedule

Tail attention (tail_attention.py, tail_op.py): behind the production FP4 indexer logits kernel and the production top-K kernel, one kernel replaces the local-to-global mapping, the sparse MLA and its reduction kernel of the unfused chain. A cluster of 5 CTAs serves a row (up to 26 rows, one wave at one CTA per SM), every CTA runs the attention of its KV tiles with a local softmax maximum and the partial outputs are combined through distributed shared memory.

FP4 megakernel: four shared selection + attention teams of 4 CTAs instead of two of 8, split ranges rotated over the stream CTAs per row (one stream CTA holds ~35 % of a row's candidates on real captures), two dedicated threshold-server CTAs, the sampling ladder in the row descriptor. FP4 megakernel_max_batch is 20.

FP4 large batch: the last piece of a split row merges its row in one selection, the schedule verdict issues its loads before its first branch, a host CTA schedule charges a CTA for every row it touches, the selector warps finalize the last segment without waiting for the row warps, faster selection and compaction loops, and a second compile of the indexer kernel from 60 rows with six K stages and a longer selector sleep.

The FP8 spec generates the same PTX / IR as before. Tests: tail attention, host schedule and knob tables (CPU), FP4 megakernel sizes 1 to 20, negative-logit regression for a signed Uint32 compaction bound, config and op switch sizes.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…ow-boundary knobs for 1 to 4 rows

The FP4 stream's sample-window index pos // wst becomes umulhi(pos, magic) (exact for pos < 2^18, 8 <= wst < 2^14; checked exhaustively on the host), which shortens the math warps' per-element path: the megakernel is 1 to 4 % faster at 1 to 20 rows. The FP4 sets of 1 to 4 rows add the existing start / row-boundary knobs (descriptor prefetch at entry and one row ahead, rotated split ranges).

README: the fastest path per batch size against the production FP4 chain.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…4 rows

The threshold lead of the FP4 megakernel started polling the sample board only after ~4.4 us because of two start fences under the start burst: the release fence now overlaps the first poll loads (lpoll, lfence at one row). The bitwise rank search becomes a 256-bin histogram with two barriers (lbin, bit-identical threshold line), the stream CTAs read the threshold line in one round trip with tagged validation (tpoll1), the classification of buffered tiles issues four tiles' loads first (cls4) and the followers' dry pass warms the threshold store block instead of racing the lead (fwarm). All knobs are default off and set only for the FP4 sets of 1 to 4 rows; the FP8 kernels and the FP4 kernels at 5 or more rows generate the same code as before.

Tests: the FP4 knob-table assertion and a test that the published threshold line does not depend on the new knobs (1 to 4 rows).

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
… rows

From 512 rows on the 1000 W board power cap lowers the SM clock from 1.63 to 1.38 GHz. The per-tile chain of the large-batch indexer kernel's math warps (about 292 SM cycles per 128 positions) is longer than the HBM time of a tile (184 ns) below 1.59 GHz, so the kernel becomes SM bound while the production indexer kernel stays HBM bound. Four knobs take instructions and hand-offs off that chain: lim_loop and lim_loop2 (the loop control is one compare against a per-segment limit), nolock (the UMMA warps meet only at a row change) and floor_mask (the floor is read less often). They are default off and set by a three-tier table: from 60 rows (sel_idle_ns 1536, kv_stages 6, lim_loop, lim_loop2, nolock), from 192 rows (plus floor_mask 30) and from 768 rows (sel_idle_ns 3072, kv_stages 8). The FP8 kernels and the FP4 kernels below 60 rows generate the same code as before.

nsys against the FP4 chain, 4 passes: +3.4 % at 1024 rows, +2.9 % at 768, +1.6 % at 512, +0.3 to +1.1 % at 64 to 384 rows; 32 to 56 rows within +-0.1 %.

Tests: the knob tables of the large-batch kernel and a test that the limit loop visits the same tiles as the shipped loop condition.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
longcheng-nv and others added 5 commits November 21, 2026 09:49
…-batch size, wide output stores

The first dispatch tier of the indexer kernel (sel_idle_ns 1536, kv_stages 6, lim_loop, lim_loop2, nolock) now starts at the first large-batch size instead of at 60 rows: the base knobs (8 K stages, sel_idle_ns 128) are 0.9 to 2 % slower at 27 to 59 rows, and 8 stages with the loop-control trims are 3 to 4 % slower at 47 to 64 rows. The output epilogue of the attention kernel writes 32-byte stores (st.global.v8, half the store instructions of 16-byte ones; the epilogue is about 3 us of the ~13 us row latency that ends every call); an output that is only 16-byte aligned runs a second, narrower compile (every torch allocation is 32-byte aligned).

nsys against the FP4 chain, 3 passes, against the previous commit: +1.0 to +2.2 % at 32 to 56 rows, within +-0.2 % at 64 to 384 rows, +0.5 to +0.6 % at 512 to 1024 rows.

Tests: the large-batch knob tables and the output alignment (8 cases for 16- and 32-byte aligned outputs).

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…8 rows

One row: the followers' dry pass of the threshold code (fwarm) is off; it measures 0.1 to 0.2 us slower in the nsys A / B of the unstamped kernel. Two and three rows: the candidate publish of the last row goes out in batches (b1l, b1l_u 3; at two rows also b1l_pub, b1l_upub 3). Five to eight rows: the threshold lead ranks the sampled logits with the 256-bin histogram (lbin, the published threshold line is bit for bit the same) and, at eight rows, the candidate publish of the last row goes out four per lane. Table changes only: no kernel code changes, the FP8 sets are untouched.

nsys fused time, 4 chunks against 2 to 3 chunks: 1 row -0.12 us, 2 rows -0.07, 3 rows -0.27, 5 rows -0.90, 6 rows -0.54, 7 rows -0.76, 8 rows -1.28; 4 rows is unchanged.

Tests: the FP4 knob-table assertion and the published-threshold-line test extended to 5 to 8 rows.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…brid

topk_multicta (topk_multicta.py, host side topk_multicta_op.py): one cluster of g CTAs per row reads each row once, a sampled threshold bins the candidates, the CTAs exchange per-bin counts through distributed shared memory and write everything above the crossing bin directly; a row that fails the count certificate takes an exact four-level radix selection over the whole row, so the result is exact for every input. tail_op.run_hybrid uses it when every row has at least 16,384 positions.

The radix fallback had a write-after-read race between its levels (found under compute-sanitizer timing, invisible to the stress runs, the gates and synccheck); one barrier per level fixes it, racecheck reports no errors and a test fuzzes the kernel schedule.

Tests: test_cute_dsl_spec_dsa_topk.py.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…i-CTA top-K (PDL)

The multi-CTA top-K executes griddepcontrol.launch_dependents after its stream phase and the tail attention is launched with the programmatic attribute. Its clusters start on the SMs the top-K leaves free and run their barrier and TMEM setup, the query load and the window tile while the top-K ranks the crossing bin; the griddepcontrol.wait in front of the first read of the selection was already there. run_hybrid(pdl=None) turns both on with the multi-CTA top-K (pdl=False keeps the plain stream order); both programmatic variants are compiled by precompile, which must run before a CUDA-graph capture. The PTX of the default-config kernels is unchanged, the trigger variant has one added instruction.

nsys against the same hybrid in plain launch order, 3 passes, 9 to 24 rows: -0.53 us on average (-0.83 %), up to -1.4 us at 10 rows and ~0 at 24 rows (the top-K holds 120 SMs there). The trigger point matters: at the kernel start +3.9 us at 9 rows, after the stream phase -0.83 %.

Tests: an ordering test with a slow stand-in top-K that triggers early (a tail attention without griddepcontrol.wait fails it) and the validation / API tests of the programmatic variants (12 new tests in each of the tail-attention and top-K test files).

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…ttention, 8-CTA top-K clusters at 13 to 15 rows, PDL up to 24 rows

The CTAs of the tail attention compute and send the 128-column output slices in the order of their rank (CTA r starts with its own slice and then serves the owners r + 1, r + 2, ... one after the other), so that every owner receives from one CTA at a time; before, all senders queued at owner 0, then at owner 1, and so on, and the distributed shared memory of an SM takes about 28 bytes per clock. The slices are sent before a CTA waits for the (max, sum) pairs of its peers; the results are bit-identical. With the programmatic launch the CTAs of the selected tiles bring their share of the block-table row into L2 while the top-K kernel finishes. The multi-CTA top-K uses clusters of 8 CTAs at 13 to 15 rows (6 before; 0.5 to 1.2 us faster behind the programmatic launch). run_hybrid launches programmatically by default up to 24 rows (tail_op.PDL_MAX_BATCH): at 25 and 26 rows the 26 clusters of one wave are all but full and the early launch costs up to 3 us. The README dispatch becomes: megakernel up to 4 rows, tail attention from 5 to 26 rows, large batch above.

nsys against the FP4 chain, 3 passes, fused time of the hybrid against the previous commit: -0.5 to -2.4 us at 5 to 24 rows (mean -1.2 us at 9 to 24 rows). From 5 rows the hybrid is 1 to 3 us faster than the megakernel; at 3 and 4 rows the megakernel stays 2.8 to 3.4 us ahead.

Tests: the default launch mode and the stage order with a slow trigger kernel at 3, 24, 25 and 26 rows.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant