Repository navigation
Add DSv4 cudnn sparse attention kernel adapters to TE - #3566
sudhakarsingh27 wants to merge 46 commits into
Conversation
DSv4 HCA and CSA need joint attention over local and compressed KV rows, while CSA also selects compressed rows. Expose the cuDNN Frontend compressor, selector, and sparse attention operations through a small parameter-free TE module so model-owned projections and positional handling remain reusable. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Build the BSD-to-BSD CSA/HCA layer over the existing cuDNN-backed compressor, indexer, and attention operations so callers can supply hidden states directly. Keep the shared RoPE preparation and indexer in small modules, and cache FP32 frequencies across forwards. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Megatron trains its CSA indexer with a separate dense objective, while discrete top-k selection leaves the indexer without language-model gradients. Expose the live indexer tensors and narrow cuDNN score/backward calls so model code can retain ownership of KL reduction and loss attachment. Allow a caller RoPE module to match compressed YaRN without copying that policy into TE; preserve the default BSD return. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Keep the DSv4 API branch mergeable as attention exports evolve upstream. Preserve both the existing linear attention exports and the DSv4 sparse attention exports in the public package. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The repository Black configuration reformats the newly added DSv4 files. Apply it so lint can assess the implementation without style-only failures; this commit changes no behavior. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
for more information, see https://pre-commit.ci
The PR Python check reports DSv4 docstring, literal, unused-argument, and local-initialization findings. Resolve them without changing the attention or indexer contracts so CI can validate the implementation. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Megatron passes sequence-major hidden states to attention. Accept and return that layout by default so the batch-one path can enter the existing packed-row core through a view without an extra public BSD round trip. Keep explicit BSD input for batch-major callers; larger batches still repack for the present core layout. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
|
A few initial comments - I know the PR is still in draft.
Thanks! |
The eager FP32 square/mean path materializes full-size query intermediates. At B4/S32768/BF16 on B200, weightless F.rms_norm lowers HCA and CSA TE peak allocated memory by about 45 GiB and cuts forward/backward time by 26-28%, while mapped-weight output and gradient cosines remain above 0.999999. Keep the unweighted normalization contract without adding a learned scale. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
cuDNN requires max_seqlen_q <= max_seqlen_k * ratio. Keep the compressed prefixes at the number of complete blocks, but pass a ceil-sized compile bound so non-divisible sequence lengths work. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Name each module after the implementation it owns, and merge the parameter-free DSv4Attention core with the high-level attention layer. Remove the redundant dsv4 wrapper and legacy alias while preserving tensor contracts, state layout, and execution order. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The module adapts PyTorch and autograd calls to cuDNN DSA APIs rather than implementing kernels, so dsa_cudnn is the shorter and more accurate boundary name. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Python-side CuTe launches dominate the otherwise fast rotary kernels. Register shape-specialized CuTe callables through the common TVM-FFI cache so steady-state SM100 execution dispatches from C++ while unsupported inputs keep the eager fallback. Compile the weightless query normalization and let its fresh output own the in-place query rotation. This removes one full-width query copy without changing the public attention or state-dict contract; other RoPE sites retain out-of-place semantics required by their autograd boundaries. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
for more information, see https://pre-commit.ci
Follow the existing TE packed-attention contract at the high-level layer so callers can pass [T,D] hidden states with CUDA cumulative lengths. Build compressed prefixes on device to avoid a per-step host synchronization, and clear unused capacity rows before normalization because cuDNN only guarantees rows below the final compressed prefix. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use the upstream forward/backward kernels from Megatron-LM c4129afff4867ca89b1fb7b4b911128424cdfbe8 while retaining TE tensor ownership and gradient semantics. Keeping the existing out-of-place KV/output and in-place fresh-query boundaries isolates the kernel-backend change and preserves autograd correctness. Remove the DSv4-specific CuTe implementation and native bridge so this branch has a single optimized RoPE backend. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Remove the parameter-free module that only stored window/ratio and delegated to the backend function. Call that function directly from the high-level layer and retain its FP8 rejection, giving DSv4Attention one clear hidden-state contract without an additional wrapper or compatibility alias. Prepared-tensor callers continue to use dsa_cudnn.attention. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use qkv_format with sbhd, bshd, and thd as in MultiheadAttention and DotProductAttention. The names describe the attention layout while high-level hidden states omit the head axis. Preserve the same dense and packed computation without aliases for the retired spelling. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Name the prepared-tensor operation sparse_attention to distinguish it from the hidden-state layer. Update callers and tests together, and document the module hierarchy and CSA/HCA usage without repeating implementation detail. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use a named launch-grid function and document the rotary autograd methods. Split packed-prefix type validation from tensor checks to satisfy the Boolean-expression limit while preserving the same validation and error message. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
5876d2c to
06013ce
Compare
[Medium impact] The latest changes appear safe to merge; no new actionable issues were found. SummaryAdds experimental
Diagram%%{init: {'theme': 'neutral'}}%%
flowchart LR
H[Hidden states] --> Q[Query projection and RoPE]
H --> U[Uncompressed KV]
H --> C[Compressor and RoPE]
H --> I[CSA Indexer]
Q --> A[cuDNN sparse attention]
U --> A
C --> A
I --> A
A --> O[Inverse RoPE and output projection]
Reviews (10) · Last reviewed commit: "Derive the HCA example slot counts" · Reviewed by Greptile |
Keep representative RoPE and packed-attention cases to avoid repeated test work. Align RoPE entry point names with their callers and export the existing loss helpers so model code can train the indexer through the public API. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Bring the branch up to date with upstream before publishing the review changes. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use one RoPE adapter and autograd function so callers can opt into reusing a disposable query buffer while preserving inputs by default. Expose the table cache for standalone sparse attention components and honor the requested device on cache hits. Keep the existing adapter parity cases and exercise cache migration in the CSA integration test. Simplify the MLA KV test name and explain how it assembles keys and values. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Share table construction and the Triton Q launch across DSA and MLA while preserving each path's layout and mutation contract. Keep standalone RoPE coverage together and zero-fill masked MLA heads so partial blocks contribute correctly to the positional-key gradient. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use ape for the learned additive position embedding in the module, cuDNN wrapper, and examples. The experimental compressor state-dict key now matches the reference implementation. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The standalone Indexer tests check numerical backward. Exercise the single-sequence context returned by the full layer to verify that the dense loss helpers train the Indexer without sending gradients to the input or other layer parameters. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Keep Q kernel attribution in the shared module. Name the forward and backward entry points in the DSv4 and MLA documentation and identify the separate KV kernel used by the MLA reference tests. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
|
/te-ci pytorch L0 |
|
Following up on the initial feedback.
|
| for actual, expected in zip(candidate, reference): | ||
| assert torch.isfinite(actual).all() and actual.float().norm() > 0 | ||
| error = (actual.float() - expected.float()).norm() / expected.float().norm().clamp_min(1e-8) | ||
| assert error < 0.03, error.item() |
There was a problem hiding this comment.
Any reasoning behind the usage of the tolerances in this test, for example, 0.03 in the two asserts above and then "atol=2e-4, rtol=2e-2" at the beginning? Thanks.
There was a problem hiding this comment.
The probability check uses the same 2% relative tolerance as TE’s fused-router softmax test, with a smaller absolute allowance near zero. For gradients, we measure error across each tensor using the same 3% relative L2 limit as an existing TE Linear gradient test. Across the 32- and 64-head cases, we measured at most 6e-8 absolute probability error and 0.33% relative L2 gradient error.
| assert param.grad is not None, name | ||
| assert torch.isfinite(param.grad).all() and torch.count_nonzero(param.grad) > 0, name | ||
| else: | ||
| assert param.grad is None, name |
There was a problem hiding this comment.
Do we only need to do any numerical checks here for the gradients?
There was a problem hiding this comment.
This test backpropagates only the Indexer auxiliary loss, so it checks that non-Indexer parameters receive no gradients. We check backward through the main attention output separately. test_dsa_sparse_attention compares gradients against a PyTorch reference. test_dsv4_attention_thd compares input and parameter gradients from one packed call with those from running each sequence individually at batch size 1.
|
|
||
| The first overlapping entry masks the missing previous block. Both modes emit | ||
| ``floor(length / ratio)`` compressed KV tokens per sequence. Trailing tokens | ||
| that do not fill a complete block (``k`` here) are excluded from compression. |
There was a problem hiding this comment.
Is this true that trailing tokens are not producing another block? Is this how DSV4 works or cuDNN compressor works? (Just checking and wanting to make sure our claim is accurate)
There was a problem hiding this comment.
Yes, this matches both DeepSeek V4 and cuDNN. Only complete blocks produce compressed entries. Trailing tokens are still available to sliding-window attention. DeepSeek’s inference implementation also buffers them so they can form a complete block during later decoding. This PR covers full-sequence execution.
| # window=ratio=2: cu_comp=[0,2,7,15]; local=[6,7]; compressed=[2,3]. | ||
| # Combined KV indices: [6,7,34,35,-1,-1,-1,-1,-1,-1]. | ||
| # window=ratio=3: cu_comp=[0,1,4,9]; local=[5,6,7]; compressed=[1]. | ||
| # Combined KV indices: [5,6,7,33,-1,-1,-1,-1]. |
There was a problem hiding this comment.
I think I'm a little confused by the example here :-) but that could be just me. Where did 33/34/35 come from? Can ping you offline for explanation as well. Thanks.
There was a problem hiding this comment.
The combined KV tensor starts with 32 uncompressed entries. Adding that offset maps [2, 3] to [34, 35] at ratio 2 and 1 to 33 in the ratio-3 illustration. I expanded the worked examples in the _prepare_sparse_kv_indices docstring.
cyanguwa
left a comment
There was a problem hiding this comment.
A few minor comments - please try to address them, and happy to re-approve! Thanks!
Match the specific metadata diagnostics so each rejection test verifies the intended validation branch. Describe packed output padding, KV offsets, and the fused projection weight order where readers use those contracts. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The full KV bank retains every input token and supplies the sliding window for each query. Calling this bank local KV obscures that distinction. Rename the sparse_attention keyword and returned Indexer context key consistently, together with their callers and documentation. Preserve checkpoint keys and backend tensor contracts. Validation: configured pre-commit hooks, whitespace checks, and Python AST equivalence after normalizing the approved names and diagnostic strings. GPU validation is pending. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Describe sequence isolation and the uncompressed-to-compressed index offset before the worked HCA and CSA examples. Include sequence boundaries and invalid slots to clarify why compressed indices cannot be interpreted as positions within the combined KV tensor. Keep the ratio-3 example explicitly separate from supported compressor ratios. Validation: configured pre-commit hooks, whitespace checks, unchanged executable AST, and checked example arithmetic. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Preserve the upstream changes already on the PR while retaining the reviewed DSv4 commits. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Explain how the longest sequence determines the compressed slot count and distinguish shorter-sequence padding from future blocks in the query example. Executable AST is unchanged and whitespace checks pass. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
|
@cyanguwa, I've addressed/clarified the concerns. Please have a look and approve again! Thanks! |
Description
Add experimental DSv4 attention (HCA/CSA) and reusable building blocks for other sparse attention layers.
Type of change
Changes
DSv4Attentionfrom TE projections/norms, publicCompressorandIndexermodules, cuDNN compression/sparse-attention forward and backward kernels, and top-k selection. Thedense_indexer_loss_scoresanddense_indexer_loss_backwardhelpers return score buffers and explicit gradients for a caller-owned auxiliary loss.sbhd,bshdand packedthdformats, fused shared-input projections, injectable RoPE embeddings and optional CSA indexer context. Current support and limitations are documented in the sparse-attention README.Validation
Checklist: