Skip to content

[PyT] [Common] Enable Context Parallelism for GDN attention - #3826

Open
KshitijLakhani wants to merge 39 commits into
NVIDIA:mainfrom
KshitijLakhani:klakhani/feat/linear_attn_with_cp
Open

KshitijLakhani wants to merge 39 commits into
NVIDIA:mainfrom
KshitijLakhani:klakhani/feat/linear_attn_with_cp

Conversation

@KshitijLakhani

@KshitijLakhani KshitijLakhani commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Add context parallelism (CP) to PyTorch GatedDeltaNetAttention, allowing long sequences and packed documents to be processed across multiple GPUs.

Each rank owns a contiguous portion of the input. TE uses cuDNN Frontend to compute local state summaries, exchanges summaries across ranks, and calculates the starting memory needed for local attention. Backward propagates state gradients across the same document boundaries.

This PR enables GDN with contiguous ownership. The shared metadata and state-composition interfaces are designed for reuse by other linear attention variants.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Add CP configuration through cp_group and a context-parallel group setter on GatedDeltaNetAttention.
  • Support THD packed documents and BSHD/SBHD dense inputs, including uneven partitions, empty documents and empty ranks. Derive local document fragments from global cu_seqlens.
  • Implement forward and backward orchestration for summary generation, all-gather, state composition and local attention. Preserve initial/final-state gradients and activation checkpointing.
  • Add a framework-independent CUDA state-composition kernel, exposed through the common C API and a PyTorch binding.
  • Cache prepared metadata across attention modules and bypass summary communication when documents are fully contained within ranks.
  • Add a documented feature-padding workaround for incorrect cuDNN Frontend state gradients observed with irregular feature dimensions.
  • Add usage documentation, runnable examples and QA coverage. Distributed tests compare outputs, states and gradients against an independent FP64 reference; C++ tests validate state composition.

Validation

Eight distributed scenarios passed at both CP=2 and CP=4 on GB200, along with 25 C++ state-composition tests. Changed-file formatting and lint checks passed; full QA Python lint encountered existing upstream tooling issues.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@KshitijLakhani KshitijLakhani self-assigned this Nov 17, 2026
@KshitijLakhani
KshitijLakhani force-pushed the klakhani/feat/linear_attn_with_cp branch from f791ea9 to 341f835 Compare November 18, 2026 19:16
@KshitijLakhani
KshitijLakhani marked this pull request as ready for review November 21, 2026 00:19
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Replace the CP design page with a user-facing linear attention guide and runnable dense and packed GDN examples.

Include the example source in the notebook and validate it in unit and distributed CI, with independent FP64 reference coverage.

Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Run end-to-end coverage at the largest available CP size among 2, 4 and 8, and remove standalone partition, validation and binding tests.

Exercise four-document, eight-head cases and the 256-dimensional summary fallback. Preserve CP/TP subgroup coverage when running on eight GPUs.

Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Select the backward fallback transition directly, explain CP test size selection, and update the tutorial link.

Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Run nine directed scenarios in isolated workers against an independent FP64 reference. Keep the scenario matrix in pytest and use seeded bounded lengths while preserving rank and document boundary cases.

Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Remove the TP+CP-only case and retain worker output only for failures. Document token slicing and simplify scenario dispatch.

Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
@KshitijLakhani
KshitijLakhani force-pushed the klakhani/feat/linear_attn_with_cp branch from 5af3bc6 to d4d62a8 Compare November 21, 2026 00:20
@KshitijLakhani

Copy link
Copy Markdown
Collaborator Author

/te-ci pytorch L0 L1

@greptile-apps

greptile-apps Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The changes affect the Gated DeltaNet distributed-attention workflow.

The PR appears safe to merge, with a non-blocking input check recommended for dense sequence offsets.

Findings

  1. P2 Dense offsets can mix documents ▶

Summary

Adds contiguous context parallelism to Gated DeltaNet attention.

  • Gated DeltaNet runs on contiguous token slices across context-parallel ranks.
  • A linear-attention guide shows GDN inputs, outputs, and distributed use.

One non-blocking hardening issue remains: dense sequence offsets should be checked against row boundaries.

Diagram

sequenceDiagram
    participant R as CP ranks
    participant C as CUDA state composition
    participant A as Local GDN attention
    R->>R: Gather forward summaries
    R->>C: Compose starting states
    C->>A: Supply local starting states
    A-->>R: Local outputs and ending states
    R->>R: Gather backward summaries
    R->>C: Compose ending-state gradients
    C->>A: Supply gradients for local backward
    A-->>R: Local input gradients
Loading

Reviews (1) · Last reviewed commit: "Rename GDN CP length generator" · Reviewed by Greptile

Comment on lines +71 to +73
if cu_seqlens.numel() != batch + 1:
raise ValueError("Dense cu_seqlens must have shape [batch_size + 1].")
return cu_seqlens, length

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Dense offsets can mix documents

The new dense path checks the size of cu_seqlens, but not whether its offsets match the batch rows. For example, a [2, 128, H, D] input accepts [0, 64, 256]. The packed cuDNN call then resets state halfway through the first row and carries the next document across the row boundary.

Reject offsets other than [0, S, 2*S, ..., B*S], as the size-one CP path already does. This would catch accidental document mixing with a clear input error.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants