Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
8ce6c55
[None][feat] DeepSeek-V4-Flash fused sparse decode kernel for B200
longcheng-nv Nov 13, 2026
8d47993
[None][fix] dsv4_sparse_decode: fence shared-memory loads before TMA …
longcheng-nv Nov 16, 2026
e1169a8
[None][perf] dsv4_sparse_decode: earlier stream start-up at batch > 1…
longcheng-nv Nov 16, 2026
9cfd6a1
[None][feat] dsv4_sparse_decode: megakernel at every batch size from …
longcheng-nv Nov 17, 2026
e0e5ee6
[None][feat] dsv4_sparse_decode: large-batch kernels and one op for b…
longcheng-nv Nov 17, 2026
575ac4e
[None][refactor] specDSA: rename dsv4_sparse_decode to spec_dsa and m…
longcheng-nv Nov 17, 2026
560783e
[None][feat] specDSA: SpecDsaConfig framework with DeepSeek-V4-Flash …
longcheng-nv Nov 18, 2026
5ea6430
[None][perf] specDSA: one 4-CTA attention team per row at 3 and 4 rows
longcheng-nv Nov 19, 2026
5027cc1
[None][perf] specDSA: megakernel up to 47 rows, aligned CTA schedule …
longcheng-nv Nov 19, 2026
b81b2eb
[None][feat] specDSA: MXFP4 indexer K cache (DS-V4-Flash-FP4)
longcheng-nv Nov 20, 2026
c15f5cb
[None][perf] specDSA FP4: tail attention for 7 to 26 rows, megakernel…
longcheng-nv Nov 20, 2026
48c8069
[None][perf] specDSA FP4 megakernel: umulhi board index and start / r…
longcheng-nv Nov 20, 2026
2311c91
[None][perf] specDSA FP4 megakernel: shorter threshold chain at 1 to …
longcheng-nv Nov 21, 2026
75ef28c
[None][perf] specDSA FP4 large batch: power-cap knobs from 60 to 1024…
longcheng-nv Nov 21, 2026
599dbbb
[None][perf] specDSA FP4 large batch: first tier from the first large…
longcheng-nv Nov 21, 2026
23d1dd7
[None][perf] specDSA FP4 megakernel: knob tables for 1 to 3 and 5 to …
longcheng-nv Nov 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: Apache-2.0 -->

# specDSA (speculative DSA)

Fused decode kernels for the DeepSeek sparse attention (DSA) of a decode step on Blackwell: the indexer logits, the
exact top-K selection, the loads of the selected KV rows and the dense attention over them. The top-K threshold is
guessed from sampled logits as in the GVR V2 top-K and checked by a count certificate; a row where the guess does not
hold takes an exact fallback, so the selection stays exact. "spec" here is that guess, not speculative decoding (MTP).

The kernels of this package are specialized for the specs of one model, DS-V4-Flash with an FP8 or an MXFP4 indexer K
cache. `config.py` defines what a spec is and which specs have kernels; the rest of this file says where the parts that
differ between models live and how to add a model.

## Files

| File | Role |
| --- | --- |
| `config.py` | `SpecDsaConfig` (the compile-time shape of a model), the specs, the support check |
| `op.py` | the interface (`prepare`, `run`, `precompile`, `error_bits`, `implementation_for`, `is_supported`); picks the implementation of a batch size |
| `megakernel_op.py`, `large_batch_op.py` | host side of each implementation (state, argument checks, engine cache); import-safe without CuTe DSL |
| `megakernel.py`, `megakernel_attention.py`, `megakernel_lib.py` | megakernel: kernel and row descriptors, attention (KV gather, MMA, softmax), PTX helpers and selection primitives |
| `large_batch.py`, `large_batch_lib.py` | large-batch kernels: indexer kernel and attention tail, selection routines and the ready queue |
| `tail_attention.py`, `tail_op.py` | tail attention: the decode sparse MLA of up to 26 rows behind any exact top-K kernel, the KV tiles of a row split over a cluster of 5 CTAs with the combine inside the cluster (host side: `run_attention`, `run_hybrid`) |

## Implementations and specs

- Megakernel: one cooperative launch per step, up to `megakernel_max_batch` rows.
- Large-batch kernels: a persistent indexer kernel and an attention kernel behind it (programmatic dependent launch),
every batch size up to `max_batch`; one compile serves all of them.

- Tail attention (decomposed hybrid): the production indexer kernel and the production top-K kernel, then one kernel
that replaces the local-to-global mapping, the sparse MLA and its reduction kernel of the unfused chain. A cluster of
5 CTAs serve a row (up to 26 rows, the clusters of 5 that fit one wave at one CTA per SM):
every CTA runs the attention of its KV tiles with a local softmax maximum, the partial outputs are combined through
distributed shared memory. Plain stream-ordered launch (the production top-K kernel never triggers a dependent launch).
Against the production FP4 chain (1M-token captures, B200) the fastest path per batch size is the megakernel up to 8 rows,
the tail attention from 9 to 26 rows and the large-batch kernels above.

| Spec | Indexer (heads x dim, K dtype) | Top-K | Compression | KV rows | Megakernel | Large batch |
| --- | --- | --- | --- | --- | --- | --- |
| DS-V4-Flash | 64 x 128, fp8 | 512 | 4 | E4M3 rows, window 128, sink | 1 to 47 rows | 48 to 1,024 rows |
| DS-V4-Flash-FP4 | 64 x 128, fp4 | 512 | 4 | E4M3 rows, window 128, sink | 1 to 20 rows | 21 to 1,024 rows |

`is_supported(device, config)` and `implementation_for(batch_size, config)` answer from this table. For a config that is
not in it `is_supported` returns `(False, reason)` and `precompile`, `prepare` and `implementation_for` raise
`NotImplementedError` (a state carries the spec it was prepared for, so `run` needs no config); the caller keeps the
existing kernels for it. Indexer shapes of other DeepSeek models are in `config.KNOWN_SHAPES` (V3.2: 64 x 128,
top-2048, no compression; V4-Pro: top-1024, 4x compression); they have no spec yet.

## What differs between models, and where it goes

| Axis | Examples | Where |
| --- | --- | --- |
| Numbers | top-K, compression ratio, heads, head dim, page sizes, window, longest row | `SpecDsaConfig` fields; the kernels define their structural constants from the spec and reject another `kernel_key` |
| Indexer K cache | fp8 or fp4 | `index_k_dtype`; a logits front end per dtype in the megakernel's stream CTAs and in the indexer kernel (fp4: block-scaled MXF4 MMA, the scale factors go to tensor memory with UTCCP; `run` takes the query scales as `q_scale`) |
| Selection | K sets the candidate buffers, the certificate bounds and the ladder of the threshold guess | the selection primitives (`*_lib.py`); tuning that depends on K lives with them |
| KV format and dense attention | MLA latent KV (V3.2), E4M3 rows with a footer scale, window and sink (V4), the CSA2 cache (V4.1) | `kv_format`; one attention module per format (today `megakernel_attention.py` and the attention part of `large_batch.py`) |
| Schedule | teams, ring depth, batch-size ranges per implementation | the knob tables of the host modules, keyed by `kernel_key`; `megakernel_max_batch` and `large_batch` of the spec |

Model names appear in `config.py` only. The code that builds a config from a model's configuration and decides when to
use these kernels belongs with the model's attention backend (`attention/backends/sparse/<family>`), not here.

## Adding a model

1. Add the fields the model needs to `SpecDsaConfig` (with validation), and a spec to `SUPPORTED_SPECS`. Do not fork a
kernel: a number becomes a field, a different layout becomes a variant selected by a field.
2. Turn the module constants of the kernels that depend on the new fields into arguments of the kernel constructors
(today they come from `DS_V4_FLASH`). The code generated for DS-V4-Flash must not change: compare the PTX and IR of
the existing spec before and after (the megakernel at every batch-size range and both large-batch kernels).
3. Add the kernel variants (attention for a new KV format, logits for a new K dtype) and register them by the config field.
4. Validate the new spec on its own: the unit tests with its shapes, the 252-case real-capture gates (decode steps of
real captures of the model, strict top-K and attention checks), a mixed-length stress, and the batch-size sweep.
5. Document the spec in the table above.
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""specDSA (speculative DSA): fused decode sparse attention of the DeepSeek DSA models for Blackwell.

The indexer logits, the exact top-K selection, the mapping to compressed-KV rows and the sparse attention over the
selected rows run in one megakernel (few rows) or in two kernels (many rows). The top-K threshold is guessed from
sampled logits as in the GVR V2 top-K and checked by a count certificate; a row where the guess does not hold takes an
exact fallback.

The kernels are specialized for a spec (:mod:`.config`); DeepSeek-V4-Flash is the one with kernels today (``README.md``
lists the specs and how to add one). Import :mod:`.op` for the host side and :mod:`.config` for the specs; nothing is
re-exported here.
"""
Loading
Loading