Skip to content

[None][feat] specDSA: MXFP4 indexer K cache (DS-V4-Flash-FP4) and tail attention for 9 to 26 rows - #22678

Draft
longcheng-nv wants to merge 16 commits into
NVIDIA:mainfrom
longcheng-nv:perf/dsv4-specdsa-fp4
Draft

longcheng-nv wants to merge 16 commits into
NVIDIA:mainfrom
longcheng-nv:perf/dsv4-specdsa-fp4

Conversation

@longcheng-nv

@longcheng-nv longcheng-nv commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

@coderabbitai summary

Description

Follow-up to #22223 (this branch is stacked on it: review the last four commits only; it is rebased onto main when #22223 merges). Adds the MXFP4 indexer K cache of DeepSeek-V4-Flash, the format the model uses, to specDSA as a second spec, DS-V4-Flash-FP4, and serves each batch size with the implementation that is fastest against the production FP4 chain:

rows implementation what runs
1 to 8 megakernel one persistent launch: K stream, selection and attention
9 to 26 tail attention the production FP4 indexer logits kernel and the production top-K kernel, then one kernel (tail_op.run_attention) that replaces the chain's local-to-global mapping, sparse MLA and reduction kernel
27 to 1,024 large batch indexer kernel (K stream + selection) and attention kernel behind it (PDL)

op.run is self-contained: megakernel up to 20 rows (megakernel_max_batch of the FP4 spec) and large-batch kernels above. The tail attention is for callers that run the indexer logits and top-K kernels themselves, as the model does (tail_op.run_hybrid).

  • FP4 front ends: K pages and queries are E2M1 with a UE8M0 scale per 32 elements (68 bytes per position) and the MMA is the block-scaled MXF4 tcgen05.mma. The K scale factors are transposed in shared memory to the layout UTCCP expects and copied to tensor memory per K tile; the query scales are copied per row. The epilogue is the FP8 one without the K scale, so the logits equal the production FP4 logits kernel bit for bit and the top-512 is exact. op.run takes the query scales as q_scale.
  • Megakernel (FP4): four shared selection + attention teams of 4 CTAs instead of two of 8 (the teams limited the throughput to ~4 us per row), the split ranges of a row rotate over the stream CTAs per row (on a layer-16 capture one stream CTA holds 436 of the 1,266 candidates of a row: the most recent tokens) and two dedicated threshold-server CTAs; the ladder of the sampled threshold comes from the row descriptor.
  • Large batch (FP4): the last piece of a split row merges its row in one selection instead of one two-way selection per piece, the schedule verdict issues its loads before its first branch, a host CTA schedule that charges a CTA for every row it touches, a selector hand-over without the wait for the row warps, a second compile of the indexer kernel from 60 rows with six K stages and a longer selector sleep, and faster selection / compaction loops.
  • Tail attention: a cluster of 5 CTAs per row (up to 26 rows, one wave) runs the attention of its KV tiles with a local softmax maximum and combines the partial outputs through distributed shared memory; the launch is a plain stream-ordered launch (the production top-K kernel never triggers a dependent launch).
  • The code generated for DS-V4-Flash (FP8) is unchanged: PTX and IR of the megakernel at 1, 2, 5, 16 and 47 rows and of both large-batch kernels are identical to [None][feat] specDSA: spec framework and fused DeepSeek-V4-Flash decode kernels on B200 (batch 1 to 1,024) #22223.

Performance, B200, clean L2 (1 GiB write + 512 MiB read before every replay), nsys kernel time of a CUDA-graph replay, mean of 4 layers of a 1M-token capture x 30 replays x 3 passes. Baseline: the production FP4 chain (FP4 paged-MQA logits, GVR V2 top-K, local-to-global, sparse MLA; TRTLLM_ENABLE_PDL=1). The FP4 data are the real FP8 captures requantized to MXFP4 with the production fused_cat_fp4. For 9 to 26 rows the time is the whole graph: production indexer logits, production top-K and the tail attention.

rows chain (us) specDSA (us) speedup implementation
1 29.4 18.0 1.629 megakernel
2 33.5 21.1 1.592 megakernel
3 37.4 26.2 1.425 megakernel
4 40.5 30.3 1.337 megakernel
5 42.7 37.0 1.153 megakernel
6 45.9 40.2 1.144 megakernel
7 49.0 44.0 1.114 megakernel
8 53.9 47.9 1.126 megakernel
10 61.4 54.0 1.136 tail attention
12 68.6 60.5 1.135 tail attention
13 79.0 64.0 1.233 tail attention
14 82.3 67.6 1.218 tail attention
16 79.5 72.6 1.095 tail attention
20 95.5 87.9 1.086 tail attention
24 105.4 97.9 1.077 tail attention
32 131.2 125.3 1.047 large batch
40 161.2 144.1 1.119 large batch
48 184.8 164.3 1.125 large batch
64 231.3 205.2 1.127 large batch
100 346.8 298.0 1.164 large batch
148 498.0 418.8 1.189 large batch
192 646.6 541.0 1.195 large batch
256 847.6 710.9 1.192 large batch
384 1257.7 1066.8 1.179 large batch
512 1670.4 1444.0 1.157 large batch
768 2520.2 2222.7 1.134 large batch
1024 3322.4 3017.7 1.101 large batch

specDSA FP4 vs the FP4 chain

Over the 40 batch sizes below the geometric mean of the speedup is 1.161 and the minimum 1.047 (at 32 rows); no size is slower than the chain. By range: 1 to 8 rows 1.300 (min 1.114), 9 to 26 rows 1.115 (min 1.074), above 26 rows 1.143 (min 1.047). The chain itself has a slow route at 13 and 14 rows (its sparse MLA takes 18 instead of 10 us), which is why the tail attention reads 1.2x there.

The K stream is HBM-bound in every path (the production indexer logits kernel runs at ~6.0 TB/s, the large-batch indexer kernel at ~6.6 TB/s), so above ~100 rows the gain is the part of the selection and attention that the large-batch kernels hide behind the stream (1.16 to 1.19x), and below 27 rows it is the shorter tail after the K stream (the chain's top-K, mapping and attention kernels take 25 to 35 us; the tail attention replaces the last two of them).

All measured batch sizes (mean over replays; p10 / p90 of the specDSA replays)
rows chain (us) specDSA (us) speedup specDSA p10 / p90 (us) implementation
1 29.4 18.0 1.629 17.7 / 18.5 megakernel
2 33.5 21.1 1.592 20.6 / 21.5 megakernel
3 37.4 26.2 1.425 25.6 / 26.8 megakernel
4 40.5 30.3 1.337 29.8 / 30.9 megakernel
5 42.7 37.0 1.153 36.4 / 37.7 megakernel
6 45.9 40.2 1.144 39.6 / 40.7 megakernel
7 49.0 44.0 1.114 43.2 / 44.7 megakernel
8 53.9 47.9 1.126 47.3 / 48.5 megakernel
9 57.1 50.6 1.128 50.1 / 51.2 tail attention
10 61.4 54.0 1.136 53.5 / 54.6 tail attention
11 64.7 57.5 1.125 57.0 / 57.9 tail attention
12 68.6 60.5 1.135 60.0 / 61.0 tail attention
13 79.0 64.0 1.233 63.5 / 64.6 tail attention
14 82.3 67.6 1.218 66.9 / 68.2 tail attention
15 76.7 69.6 1.103 69.1 / 70.1 tail attention
16 79.5 72.6 1.095 72.1 / 73.2 tail attention
17 84.2 77.3 1.090 76.5 / 78.0 tail attention
18 87.7 80.2 1.094 79.6 / 80.8 tail attention
19 90.3 83.2 1.085 82.4 / 83.9 tail attention
20 95.5 87.9 1.086 87.2 / 88.7 tail attention
21 97.0 89.3 1.086 88.7 / 90.0 tail attention
22 100.6 93.6 1.074 92.8 / 94.5 tail attention
23 102.3 94.4 1.083 93.8 / 95.0 tail attention
24 105.4 97.9 1.077 97.1 / 98.6 tail attention
32 131.2 125.3 1.047 123.6 / 126.8 large batch
40 161.2 144.1 1.119 142.8 / 145.6 large batch
47 180.5 162.1 1.113 160.4 / 163.8 large batch
48 184.8 164.3 1.125 163.0 / 165.7 large batch
56 208.8 186.0 1.123 184.3 / 187.5 large batch
64 231.3 205.2 1.127 203.7 / 206.6 large batch
80 286.2 245.7 1.165 244.3 / 247.4 large batch
100 346.8 298.0 1.164 296.1 / 300.1 large batch
128 434.2 373.2 1.163 370.6 / 376.2 large batch
148 498.0 418.8 1.189 416.9 / 420.8 large batch
192 646.6 541.0 1.195 538.0 / 544.6 large batch
256 847.6 710.9 1.192 707.4 / 716.6 large batch
384 1257.7 1066.8 1.179 1049.2 / 1094.7 large batch
512 1670.4 1444.0 1.157 1398.3 / 1516.9 large batch
768 2520.2 2222.7 1.134 2090.8 / 2335.0 large batch
1024 3322.4 3017.7 1.101 2803.1 / 3149.7 large batch

Test Coverage

Unit tests (B200), tests/unittest/_torch/attention/sparse/deepseek_v4/:

  • test_cute_dsl_spec_dsa_tail.py (new): 155 passed; tail attention against the FP32 reference at the cluster boundaries, short windows, -1 padded and empty selections, CUDA-graph replay, error cases, and a power-of-two mode that checks the cross-CTA combine at 2e-5.
  • test_cute_dsl_spec_dsa_host.py (new): 134 passed (CPU only): host CTA schedule invariants, the FP8 schedule equal to the previous one on 2,324 (batch, length) mixes, knob tables.
  • test_cute_dsl_spec_dsa_large_batch.py: 48 passed (FP8 and FP4 fixtures, mixed lengths, ties, all-equal rows, state reuse, CUDA-graph replay, negative logits: a regression test for a signed min/max of a Uint32 compaction bound).
  • test_cute_dsl_spec_dsa_megakernel.py: 72 passed (FP4 cases for every size from 1 to 20).
  • test_cute_dsl_spec_dsa_op.py: 91 passed; test_cute_dsl_spec_dsa_config.py: 27 passed.

Also run on this branch (head 2311c91; the last commit changes the FP4 megakernel only, the large-batch and tail-attention runs below are from the previous commit, whose code is unchanged): the 252 real-capture decode steps (2 captures x 21 layers x 3 steps, 2 repeats; exact top-512 set against the chain's logits, attention against the FP32 reference) at 1, 2, 3, 4, 5, 6, 7, 8, 12, 16, 20, 21, 64 rows through the op and at 8, 16, 24, 26 rows through the tail attention: 4,536 of 4,536 runs pass (max calc_diff 6.51e-5); a random mixed-length stress with planted ties, all-equal rows and all-negative logits: 890 iterations (1,780 launches), 0 failures; compute-sanitizer --tool synccheck on the megakernel at 1 row and on the tail kernel at 1 and 2 rows: 0 errors (runs at 2 or more rows of the megakernel did not finish under the tool).

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

longcheng-nv and others added 10 commits November 19, 2026 14:28
Add tensorrt_llm/_torch/attention/kernels/blackwell/dsv4_sparse_decode, a
CuTe DSL kernel that runs the DeepSeek-V4-Flash CSA decode step (FP8 indexer
logits, exact top-512, KV index mapping and sparse MLA with the attention
sink) as one cooperative launch over the 148 SMs of a B200 for 1, 2, 4 or 8
rows, with its unit test. The kernel is not wired into the model yet.

Made-with: Claude Code (Opus 5.5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…stage release

The stream math warps read each K stage's per-token scales, and at batch > 1 the row's weights from the Q / W stage, and
then release the stage with an mbarrier arrive. The arrive does not order the warp's outstanding shared-memory loads
before the TMA refill (async proxy), so a delayed load could read the next tile's data. Add fence.proxy.async.shared::cta
before each of these releases (5 sites).

- Without the fences no failure was seen in 668,002 checked launches. With extra co-resident load / store traffic
  (stress-only builds), the K-stage race changed the top-512 in up to 99 % of launches.
- With the fences, 0 of 378,000 mixed-length stress launches failed, 252,000 of them under that traffic. The gates (252
  real cases per batch size) and the unit test pass.
- Cost (nsys, clean L2): +0.31 / +0.02 / +0.02 / +0.30 us at batch 1 / 2 / 4 / 8.

Made-with: Claude Code (Opus 5.5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…, batched candidate publish at batch 8

- Batch > 1: the stream CTAs' TMA warps take row 0's descriptor from the kernel-entry load and prefetch the tensor
  maps at entry, both before the CTA barrier; at batch 2 the team CTA also loads the row and window lengths before its
  barrier setup.
- Batch 8: the candidate publish issues its histogram atomics in batches of two per thread, and the last row issues its
  first batch before the count barrier. This relies on the previous commit's fences.

Fused-kernel time against the previous commit (nsys, clean L2): -0.12 / -0.04 / -1.13 us at batch 2 / 4 / 8
(a second GPU: -0.27 / -1.18 us at batch 2 / 8); batch 1 is unchanged. The gates (252 real cases per batch size) and
the unit test pass, and 0 of 136,500 stress launches (mixed-length and hot-row cases) failed.

Made-with: Claude Code (Opus 5.5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…1 to 48

The megakernel served 1, 2, 4 and 8 rows. It now serves every batch size from 1 to 48.

- 3, 5, 6 and 7 rows use the 4- and 8-row configurations.
- 9 to 16 rows: the logits of a row live in a ring of 8 shared-memory slots. A slot is reused after
  its row's selection reached K_DONE, which the T poller (warp 12) or the publishers observe before
  the row that needs the slot.
- 17 to 32 rows: a 32-entry per-row service state past the stream layout, slot-free barrier phases
  (r >> 3) - 1 & 1, up to 16 passes per team.
- 33 to 48 rows: one generation word and five state words per row, and the compressed-KV block-table
  slices of the stream CTAs as a ring of 32 entries, each reusable entry guarded by an mbarrier the
  publishers arrive on; up to 24 passes per team. The shared-memory arena stays at 231,424 bytes.
- op.py becomes megakernel_op.py. The test covers 3 to 48 rows, rows with ties, flat rows and mixed
  lengths on both sides of 32 rows.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…atch 1 to 1,024

Adds the large-batch kernels as dsv4_sparse_decode/large_batch: a persistent indexer kernel (one CTA
per SM, row segments in equal cost units, exact top-512 selection, ready queue) and an attention
kernel launched behind it with programmatic dependent launch. One compile serves every batch size.

The new op.py serves every batch size from 1 to 1,024: the megakernel up to 48 rows, the
large-batch kernels above. prepare / run / precompile / error_bits take the same arguments for both;
implementation_for names the one a batch size uses. Tests for the large-batch kernels and for the
op (every batch size around the switch, in-place refresh, CUDA graph, argument checks).

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…erge its 17 modules into 9 files

The package is the specDSA (speculative DSA) megakernel: indexer logits, top-512 selection with
a threshold guessed from sampled logits (GVR V2), compressed-KV row load and attention in one
launch up to 48 rows, in two kernels above.

- dsv4_sparse_decode -> spec_dsa; the Dsv4SparseDecode* classes -> SpecDsa*.
- megakernel: megakernel.py (kernel, row descriptors, selection entry points),
  megakernel_attention.py (attention engine and roles), megakernel_lib.py (PTX helpers,
  selection primitives, L2 policy), megakernel_op.py (host side).
- large batch: large_batch.py (attention helpers, attention tail, indexer kernel),
  large_batch_lib.py (selection routines, L2 policy), large_batch_op.py (host side).
- op.py stays the interface; the tests are test_cute_dsl_spec_dsa_{megakernel,large_batch,op}.py.
- Every line of code is kept. Only intra-package imports, re-export aliases and duplicate
  definitions are removed, and the few names that two merged modules define differently get a
  suffix. The PTX and IR of the megakernel (13 batch sizes) and of both large-batch kernels are
  identical to the previous layout.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…as its first spec

specDSA kernels are specialized for the shape of one DSA model. This adds the layer that says
which: spec_dsa/config.py defines SpecDsaConfig (indexer heads, head dim, K-cache dtype and page
size, compression ratio, top-K, longest row, attention heads, KV row layout, window, sink, and
the batch-size limits of the megakernel and the large-batch kernels) and DS_V4_FLASH, the one
spec with kernels.

- The structural constants of the kernels (top-K, heads, KV row size, window, selected rows,
  longest row, index page size, SM count) are defined from DS_V4_FLASH. The kernel constructors
  take a config and reject another kernel_key (require_built_for).
- op.py, megakernel_op.py and large_batch_op.py take an optional config (default DS-V4-Flash) in
  is_supported, precompile, prepare and implementation_for; a state keeps the spec it was
  prepared for; the knob tables are keyed by kernel_key; the module constants MIN_CTX_LEN,
  MAX_CTX_LEN, MAX_BATCH_SIZE, MEGAKERNEL_MAX_BATCH and the like are read from the spec. A
  config without kernels gives is_supported -> (False, reason) and NotImplementedError from
  precompile, prepare and implementation_for.
- README.md: what differs between models, where it goes, how a model is added.
- test_cute_dsl_spec_dsa_config.py: 25 tests that need no GPU; the three other test files take
  their shapes from the spec.
- The PTX and IR of the megakernel (13 batch sizes) and of both large-batch kernels are
  identical to the previous commit.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
Batch 3 and 4 ran on two shared teams of 8 CTAs in several passes. Each row now gets its
own team of 4 CTAs (two row groups x two V column halves of 256, one KV gather per tile),
so 3 and 4 rows run in a single pass.

- megakernel.py, megakernel_attention.py: knob team4 (team size, N = 256 PV MMA with the
  O accumulator at columns 256-511, the partial O of the partner row group handed over as
  BF16 through DSMEM: _merge_22). Off for every other batch size.
- megakernel_op.py: knob set _BS4T (team4, 4 teams, no dry merge, no candidate-row L2
  prefetch) for 3 and 4 rows; a note that compute-sanitizer synccheck only completes for 1
  to 4 rows, because from 5 rows on every pass after the first re-initializes its mbarriers
  without invalidating them.
- tests: two more 3 and 4-row cases in the megakernel test, 4 rows in the op test.

The PTX and IR of the megakernel at 1, 2, 5, 8, 9, 16, 17, 32, 33 and 40 rows and of both
large-batch kernels are identical to the previous commit. 3 and 4 rows: 1.229x and 1.231x
-> 1.333x and 1.270x of the production chain (nsys, CUDA graph, clean L2, 4 real-capture
layers, two processes). Real-capture gates 252/252 at 3 and 4 rows, synccheck clean.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…below 80 rows

The megakernel is built for up to 48 rows, but at 48 rows the large-batch kernels are
faster once their CTA schedule matches the rows.

- config.py: DS-V4-Flash megakernel_max_batch 48 -> 47; the knob table, README and
  docstrings follow.
- large_batch_op.py: below 80 rows the CTA schedule cuts each row into the fewest equal
  pieces that fit one per CTA (rows shorter than 128 units stay whole) when that gives fewer
  selection pieces than the even split and costs the busiest CTA at most 3% more stream
  work. 77 us of host time per call instead of 18 us; unchanged from 80 rows on.
- tests: an independent reference of the aligned schedule, the 47 and 48-row boundary in
  the config, op and megakernel tests.

The PTX and IR of the large-batch kernels are unchanged. Real-capture gates 252/252 at 48,
49, 56, 60, 64 and 72 rows. 48 and 72 rows: 1.047x and 1.076x -> 1.082x and 1.105x of the
production chain (nsys, CUDA graph, clean L2, 4 real-capture layers, two or more
processes); 48 to 79 rows 0.4% geometric mean; in a paired run 56 and 60 rows were about 2%
slower.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
Adds the MXFP4 indexer K cache of DeepSeek-V4-Flash to specDSA as a second
spec, DS-V4-Flash-FP4.

- FP4 logits front end in the megakernel's stream CTAs and in the large-batch
  indexer kernel: E2M1 K pages and queries with a UE8M0 scale per 32 elements,
  block-scaled MXF4 MMA, the K and query scale factors copied to tensor memory
  with UTCCP; the epilogue is the FP8 one without the K scale, so the logits
  equal the production FP4 K1 kernel.
- op.run takes the query scales as q_scale.
- Megakernel for 1 to 22 rows, large-batch kernels from 23 rows.
- The code generated for DS-V4-Flash is unchanged (PTX and IR compared).

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
… teams, large-batch merge and schedule

Tail attention (tail_attention.py, tail_op.py): behind the production FP4 indexer logits kernel and the production top-K kernel, one kernel replaces the local-to-global mapping, the sparse MLA and its reduction kernel of the unfused chain. A cluster of 5 CTAs serves a row (up to 26 rows, one wave at one CTA per SM), every CTA runs the attention of its KV tiles with a local softmax maximum and the partial outputs are combined through distributed shared memory.

FP4 megakernel: four shared selection + attention teams of 4 CTAs instead of two of 8, split ranges rotated over the stream CTAs per row (one stream CTA holds ~35 % of a row's candidates on real captures), two dedicated threshold-server CTAs, the sampling ladder in the row descriptor. FP4 megakernel_max_batch is 20.

FP4 large batch: the last piece of a split row merges its row in one selection, the schedule verdict issues its loads before its first branch, a host CTA schedule charges a CTA for every row it touches, the selector warps finalize the last segment without waiting for the row warps, faster selection and compaction loops, and a second compile of the indexer kernel from 60 rows with six K stages and a longer selector sleep.

The FP8 spec generates the same PTX / IR as before. Tests: tail attention, host schedule and knob tables (CPU), FP4 megakernel sizes 1 to 20, negative-logit regression for a signed Uint32 compaction bound, config and op switch sizes.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
@longcheng-nv longcheng-nv changed the title [None][feat] specDSA: MXFP4 indexer K cache (DS-V4-Flash-FP4) [None][feat] specDSA: MXFP4 indexer K cache (DS-V4-Flash-FP4) and tail attention for 7 to 26 rows Nov 20, 2026
…ow-boundary knobs for 1 to 4 rows

The FP4 stream's sample-window index pos // wst becomes umulhi(pos, magic) (exact for pos < 2^18, 8 <= wst < 2^14; checked exhaustively on the host), which shortens the math warps' per-element path: the megakernel is 1 to 4 % faster at 1 to 20 rows. The FP4 sets of 1 to 4 rows add the existing start / row-boundary knobs (descriptor prefetch at entry and one row ahead, rotated split ranges).

README: the fastest path per batch size against the production FP4 chain.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
@longcheng-nv longcheng-nv changed the title [None][feat] specDSA: MXFP4 indexer K cache (DS-V4-Flash-FP4) and tail attention for 7 to 26 rows [None][feat] specDSA: MXFP4 indexer K cache (DS-V4-Flash-FP4) and tail attention for 9 to 26 rows Nov 20, 2026
…4 rows

The threshold lead of the FP4 megakernel started polling the sample board only after ~4.4 us because of two start fences under the start burst: the release fence now overlaps the first poll loads (lpoll, lfence at one row). The bitwise rank search becomes a 256-bin histogram with two barriers (lbin, bit-identical threshold line), the stream CTAs read the threshold line in one round trip with tagged validation (tpoll1), the classification of buffered tiles issues four tiles' loads first (cls4) and the followers' dry pass warms the threshold store block instead of racing the lead (fwarm). All knobs are default off and set only for the FP4 sets of 1 to 4 rows; the FP8 kernels and the FP4 kernels at 5 or more rows generate the same code as before.

Tests: the FP4 knob-table assertion and a test that the published threshold line does not depend on the new knobs (1 to 4 rows).

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
longcheng-nv and others added 3 commits November 21, 2026 05:56
… rows

From 512 rows on the 1000 W board power cap lowers the SM clock from 1.63 to 1.38 GHz. The per-tile chain of the large-batch indexer kernel's math warps (about 292 SM cycles per 128 positions) is longer than the HBM time of a tile (184 ns) below 1.59 GHz, so the kernel becomes SM bound while the production indexer kernel stays HBM bound. Four knobs take instructions and hand-offs off that chain: lim_loop and lim_loop2 (the loop control is one compare against a per-segment limit), nolock (the UMMA warps meet only at a row change) and floor_mask (the floor is read less often). They are default off and set by a three-tier table: from 60 rows (sel_idle_ns 1536, kv_stages 6, lim_loop, lim_loop2, nolock), from 192 rows (plus floor_mask 30) and from 768 rows (sel_idle_ns 3072, kv_stages 8). The FP8 kernels and the FP4 kernels below 60 rows generate the same code as before.

nsys against the FP4 chain, 4 passes: +3.4 % at 1024 rows, +2.9 % at 768, +1.6 % at 512, +0.3 to +1.1 % at 64 to 384 rows; 32 to 56 rows within +-0.1 %.

Tests: the knob tables of the large-batch kernel and a test that the limit loop visits the same tiles as the shipped loop condition.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…-batch size, wide output stores

The first dispatch tier of the indexer kernel (sel_idle_ns 1536, kv_stages 6, lim_loop, lim_loop2, nolock) now starts at the first large-batch size instead of at 60 rows: the base knobs (8 K stages, sel_idle_ns 128) are 0.9 to 2 % slower at 27 to 59 rows, and 8 stages with the loop-control trims are 3 to 4 % slower at 47 to 64 rows. The output epilogue of the attention kernel writes 32-byte stores (st.global.v8, half the store instructions of 16-byte ones; the epilogue is about 3 us of the ~13 us row latency that ends every call); an output that is only 16-byte aligned runs a second, narrower compile (every torch allocation is 32-byte aligned).

nsys against the FP4 chain, 3 passes, against the previous commit: +1.0 to +2.2 % at 32 to 56 rows, within +-0.2 % at 64 to 384 rows, +0.5 to +0.6 % at 512 to 1024 rows.

Tests: the large-batch knob tables and the output alignment (8 cases for 16- and 32-byte aligned outputs).

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>
…8 rows

One row: the followers' dry pass of the threshold code (fwarm) is off; it measures 0.1 to 0.2 us slower in the nsys A / B of the unstamped kernel. Two and three rows: the candidate publish of the last row goes out in batches (b1l, b1l_u 3; at two rows also b1l_pub, b1l_upub 3). Five to eight rows: the threshold lead ranks the sampled logits with the 256-bin histogram (lbin, the published threshold line is bit for bit the same) and, at eight rows, the candidate publish of the last row goes out four per lane. Table changes only: no kernel code changes, the FP8 sets are untouched.

nsys fused time, 4 chunks against 2 to 3 chunks: 1 row -0.12 us, 2 rows -0.07, 3 rows -0.27, 5 rows -0.90, 6 rows -0.54, 7 rows -0.76, 8 rows -1.28; 4 rows is unchanged.

Tests: the FP4 knob-table assertion and the published-threshold-line test extended to 5 to 8 rows.

Made-with: Claude Code (Sonnet 5.5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant