Skip to content

Add DSv4 cudnn sparse attention kernel adapters to TE - #3566

Open
sudhakarsingh27 wants to merge 46 commits into
NVIDIA:mainfrom
sudhakarsingh27:sudhakars/dsv4-cudnn-wrapper
Open

sudhakarsingh27 wants to merge 46 commits into
NVIDIA:mainfrom
sudhakarsingh27:sudhakars/dsv4-cudnn-wrapper

Conversation

@sudhakarsingh27

@sudhakarsingh27 sudhakarsingh27 commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Description

Add experimental DSv4 attention (HCA/CSA) and reusable building blocks for other sparse attention layers.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Compose DSv4Attention from TE projections/norms, public Compressor and Indexer modules, cuDNN compression/sparse-attention forward and backward kernels, and top-k selection. The dense_indexer_loss_scores and dense_indexer_loss_backward helpers return score buffers and explicit gradients for a caller-owned auxiliary loss.
  • Support hidden states in sbhd, bshd and packed thd formats, fused shared-input projections, injectable RoPE embeddings and optional CSA indexer context. Current support and limitations are documented in the sparse-attention README.
  • Consolidate shared Megatron-derived Triton RoPE kernels for DSA and existing MLA callers; add API documentation, numerical/backward tests and the DSv4 QA test entry.

Validation

  • Exact-source native build on B200 with cuDNN Frontend 1.30.0: 35 DSv4 tests and 21 selected MLA tests passed, with no failures or skips. Performance and multi-GPU cases were deselected.
  • Native TE forward/backward canary and documented index examples passed.
  • Local formatting/pre-commit checks and pylint passed.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

DSv4 HCA and CSA need joint attention over local and compressed KV rows, while CSA also selects compressed rows. Expose the cuDNN Frontend compressor, selector, and sparse attention operations through a small parameter-free TE module so model-owned projections and positional handling remain reusable.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Build the BSD-to-BSD CSA/HCA layer over the existing cuDNN-backed compressor, indexer, and attention operations so callers can supply hidden states directly. Keep the shared RoPE preparation and indexer in small modules, and cache FP32 frequencies across forwards.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@sudhakarsingh27 sudhakarsingh27 changed the title Add DSv4 sparse attention bindings to TE Add DSv4 sparse attention layers to TE Oct 1, 2026
sudhakarsingh27 and others added 6 commits October 1, 2026 13:04
Megatron trains its CSA indexer with a separate dense objective, while discrete top-k selection leaves the indexer without language-model gradients. Expose the live indexer tensors and narrow cuDNN score/backward calls so model code can retain ownership of KL reduction and loss attachment. Allow a caller RoPE module to match compressed YaRN without copying that policy into TE; preserve the default BSD return.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Keep the DSv4 API branch mergeable as attention exports evolve upstream. Preserve both the existing linear attention exports and the DSv4 sparse attention exports in the public package.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The repository Black configuration reformats the newly added DSv4 files. Apply it so lint can assess the implementation without style-only failures; this commit changes no behavior.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The PR Python check reports DSv4 docstring, literal, unused-argument, and local-initialization findings. Resolve them without changing the attention or indexer contracts so CI can validate the implementation.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Megatron passes sequence-major hidden states to attention. Accept and return that layout by default so the batch-one path can enter the existing packed-row core through a view without an extra public BSD round trip. Keep explicit BSD input for batch-major callers; larger batches still repack for the present core layout.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@cyanguwa

cyanguwa commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

A few initial comments - I know the PR is still in draft.

  • Should we rename DSv4Attention to DSv4CoreAttention, and DSv4HybridAttention to DSv4Attention? Also, maybe we can tighten up the file naming/structure a bit more so it's more obvious to users what modules we're trying to offer. I feel maybe these ones (without the v4 denotation, given that they can be used by v4.1, Qwen as well): ds_rope, ds_compressor, ds_indexer, ds_core_attention, ds_attention?
  • You've included some explanation in README.md about the structure/usage of this PR, but I feel it's still a bit confusing. Maybe we could reuse the sketch in Po's presentation, and replace certain components with our module names?
  • How easy is it to add your Megatron/Hugging Face validator to our tests/? It could be a good way to show the usage of our PR, but this can be punted to a future PR as well (as we discussed offline) if it requires too many changes.

Thanks!

@sudhakarsingh27 sudhakarsingh27 changed the title Add DSv4 sparse attention layers to TE Add DSv4 cudnn sparse attention kernel adapters to TE Oct 7, 2026
sudhakarsingh27 and others added 12 commits October 13, 2026 11:50
The eager FP32 square/mean path materializes full-size query intermediates. At B4/S32768/BF16 on B200, weightless F.rms_norm lowers HCA and CSA TE peak allocated memory by about 45 GiB and cuts forward/backward time by 26-28%, while mapped-weight output and gradient cosines remain above 0.999999. Keep the unweighted normalization contract without adding a learned scale.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
cuDNN requires max_seqlen_q <= max_seqlen_k * ratio. Keep the compressed prefixes at the number of complete blocks, but pass a ceil-sized compile bound so non-divisible sequence lengths work.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Name each module after the implementation it owns, and merge the parameter-free DSv4Attention core with the high-level attention layer. Remove the redundant dsv4 wrapper and legacy alias while preserving tensor contracts, state layout, and execution order.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The module adapts PyTorch and autograd calls to cuDNN DSA APIs rather than implementing kernels, so dsa_cudnn is the shorter and more accurate boundary name.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Python-side CuTe launches dominate the otherwise fast rotary kernels. Register shape-specialized CuTe callables through the common TVM-FFI cache so steady-state SM100 execution dispatches from C++ while unsupported inputs keep the eager fallback.

Compile the weightless query normalization and let its fresh output own the in-place query rotation. This removes one full-width query copy without changing the public attention or state-dict contract; other RoPE sites retain out-of-place semantics required by their autograd boundaries.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Follow the existing TE packed-attention contract at the high-level layer so callers can pass [T,D] hidden states with CUDA cumulative lengths. Build compressed prefixes on device to avoid a per-step host synchronization, and clear unused capacity rows before normalization because cuDNN only guarantees rows below the final compressed prefix.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use the upstream forward/backward kernels from Megatron-LM c4129afff4867ca89b1fb7b4b911128424cdfbe8 while retaining TE tensor ownership and gradient semantics. Keeping the existing out-of-place KV/output and in-place fresh-query boundaries isolates the kernel-backend change and preserves autograd correctness. Remove the DSv4-specific CuTe implementation and native bridge so this branch has a single optimized RoPE backend.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Remove the parameter-free module that only stored window/ratio and delegated to the backend function. Call that function directly from the high-level layer and retain its FP8 rejection, giving DSv4Attention one clear hidden-state contract without an additional wrapper or compatibility alias. Prepared-tensor callers continue to use dsa_cudnn.attention.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use qkv_format with sbhd, bshd, and thd as in MultiheadAttention and DotProductAttention. The names describe the attention layout while high-level hidden states omit the head axis. Preserve the same dense and packed computation without aliases for the retired spelling.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Name the prepared-tensor operation sparse_attention to distinguish it from the hidden-state layer. Update callers and tests together, and document the module hierarchy and CSA/HCA usage without repeating implementation detail.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use a named launch-grid function and document the rotary autograd methods. Split packed-prefix type validation from tensor checks to satisfy the Boolean-expression limit while preserving the same validation and error message.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@sudhakarsingh27
sudhakarsingh27 force-pushed the sudhakars/dsv4-cudnn-wrapper branch from 5876d2c to 06013ce Compare October 13, 2026 18:50
@sudhakarsingh27
sudhakarsingh27 marked this pull request as ready for review October 13, 2026 18:51
@greptile-apps

greptile-apps Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The latest changes appear safe to merge; no new actionable issues were found.

Summary

Adds experimental DSv4Attention, reusable Compressor and Indexer modules, and cuDNN sparse-attention adapters.

  • DSv4 turns hidden states into HCA or CSA sparse-attention outputs.
  • CSA can return indexer data for a caller-owned training loss.
  • DSv4 and MLA use shared rotary-position kernels and helpers.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  H[Hidden states] --> Q[Query projection and RoPE]
  H --> U[Uncompressed KV]
  H --> C[Compressor and RoPE]
  H --> I[CSA Indexer]
  Q --> A[cuDNN sparse attention]
  U --> A
  C --> A
  I --> A
  A --> O[Inverse RoPE and output projection]
Loading

Reviews (10) · Last reviewed commit: "Derive the HCA example slot counts" · Reviewed by Greptile

Keep representative RoPE and packed-attention cases to avoid repeated test work. Align RoPE entry point names with their callers and export the existing loss helpers so model code can train the indexer through the public API.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Bring the branch up to date with upstream before publishing the review changes.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use one RoPE adapter and autograd function so callers can opt into reusing a disposable query buffer while preserving inputs by default. Expose the table cache for standalone sparse attention components and honor the requested device on cache hits.

Keep the existing adapter parity cases and exercise cache migration in the CSA integration test. Simplify the MLA KV test name and explain how it assembles keys and values.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Share table construction and the Triton Q launch across DSA and MLA while preserving each path's layout and mutation contract. Keep standalone RoPE coverage together and zero-fill masked MLA heads so partial blocks contribute correctly to the positional-key gradient.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use ape for the learned additive position embedding in the module, cuDNN wrapper, and examples. The experimental compressor state-dict key now matches the reference implementation.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The standalone Indexer tests check numerical backward. Exercise the single-sequence context returned by the full layer to verify that the dense loss helpers train the Indexer without sending gradients to the input or other layer parameters.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Keep Q kernel attribution in the shared module. Name the forward and backward entry points in the DSv4 and MLA documentation and identify the separate KV kernel used by the MLA reference tests.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@sudhakarsingh27

Copy link
Copy Markdown
Member Author

/te-ci pytorch L0

@sudhakarsingh27

Copy link
Copy Markdown
Member Author

Following up on the initial feedback.

DSv4Attention is the hidden-state layer, and dsa_cudnn.sparse_attention is the entry point for prepared tensors. The README shows the module hierarchy and data flow, with CSA and HCA examples. Megatron and Hugging Face integration tests can follow in a separate PR.

Comment thread tests/pytorch/test_dsv4_attention.py Resolved
for actual, expected in zip(candidate, reference):
assert torch.isfinite(actual).all() and actual.float().norm() > 0
error = (actual.float() - expected.float()).norm() / expected.float().norm().clamp_min(1e-8)
assert error < 0.03, error.item()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reasoning behind the usage of the tolerances in this test, for example, 0.03 in the two asserts above and then "atol=2e-4, rtol=2e-2" at the beginning? Thanks.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The probability check uses the same 2% relative tolerance as TE’s fused-router softmax test, with a smaller absolute allowance near zero. For gradients, we measure error across each tensor using the same 3% relative L2 limit as an existing TE Linear gradient test. Across the 32- and 64-head cases, we measured at most 6e-8 absolute probability error and 0.33% relative L2 gradient error.

assert param.grad is not None, name
assert torch.isfinite(param.grad).all() and torch.count_nonzero(param.grad) > 0, name
else:
assert param.grad is None, name

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we only need to do any numerical checks here for the gradients?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test backpropagates only the Indexer auxiliary loss, so it checks that non-Indexer parameters receive no gradients. We check backward through the main attention output separately. test_dsa_sparse_attention compares gradients against a PyTorch reference. test_dsv4_attention_thd compares input and parameter gradients from one packed call with those from running each sequence individually at batch size 1.


The first overlapping entry masks the missing previous block. Both modes emit
``floor(length / ratio)`` compressed KV tokens per sequence. Trailing tokens
that do not fill a complete block (``k`` here) are excluded from compression.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this true that trailing tokens are not producing another block? Is this how DSV4 works or cuDNN compressor works? (Just checking and wanting to make sure our claim is accurate)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this matches both DeepSeek V4 and cuDNN. Only complete blocks produce compressed entries. Trailing tokens are still available to sliding-window attention. DeepSeek’s inference implementation also buffers them so they can form a complete block during later decoding. This PR covers full-sequence execution.

# window=ratio=2: cu_comp=[0,2,7,15]; local=[6,7]; compressed=[2,3].
# Combined KV indices: [6,7,34,35,-1,-1,-1,-1,-1,-1].
# window=ratio=3: cu_comp=[0,1,4,9]; local=[5,6,7]; compressed=[1].
# Combined KV indices: [5,6,7,33,-1,-1,-1,-1].

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'm a little confused by the example here :-) but that could be just me. Where did 33/34/35 come from? Can ping you offline for explanation as well. Thanks.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The combined KV tensor starts with 32 uncompressed entries. Adding that offset maps [2, 3] to [34, 35] at ratio 2 and 1 to 33 in the ratio-3 illustration. I expanded the worked examples in the _prepare_sparse_kv_indices docstring.

cyanguwa
cyanguwa previously approved these changes Nov 20, 2026

@cyanguwa cyanguwa left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor comments - please try to address them, and happy to re-approve! Thanks!

Match the specific metadata diagnostics so each rejection test verifies the intended validation branch. Describe packed output padding, KV offsets, and the fused projection weight order where readers use those contracts.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The full KV bank retains every input token and supplies the sliding window for each query. Calling this bank local KV obscures that distinction. Rename the sparse_attention keyword and returned Indexer context key consistently, together with their callers and documentation. Preserve checkpoint keys and backend tensor contracts.

Validation: configured pre-commit hooks, whitespace checks, and Python AST equivalence after normalizing the approved names and diagnostic strings. GPU validation is pending.
Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Describe sequence isolation and the uncompressed-to-compressed index offset before the worked HCA and CSA examples. Include sequence boundaries and invalid slots to clarify why compressed indices cannot be interpreted as positions within the combined KV tensor. Keep the ratio-3 example explicitly separate from supported compressor ratios.

Validation: configured pre-commit hooks, whitespace checks, unchanged executable AST, and checked example arithmetic.
Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Preserve the upstream changes already on the PR while retaining the reviewed DSv4 commits.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Explain how the longest sequence determines the compressed slot count and distinguish shorter-sequence padding from future blocks in the query example. Executable AST is unchanged and whitespace checks pass.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@sudhakarsingh27

Copy link
Copy Markdown
Member Author

@cyanguwa, I've addressed/clarified the concerns. Please have a look and approve again! Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants