Repository navigation
[PyT] [Common] Enable Context Parallelism for GDN attention - #3826
KshitijLakhani wants to merge 39 commits into
Conversation
f791ea9 to
341f835
Compare
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Replace the CP design page with a user-facing linear attention guide and runnable dense and packed GDN examples. Include the example source in the notebook and validate it in unit and distributed CI, with independent FP64 reference coverage. Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Run end-to-end coverage at the largest available CP size among 2, 4 and 8, and remove standalone partition, validation and binding tests. Exercise four-document, eight-head cases and the 256-dimensional summary fallback. Preserve CP/TP subgroup coverage when running on eight GPUs. Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Select the backward fallback transition directly, explain CP test size selection, and update the tutorial link. Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Run nine directed scenarios in isolated workers against an independent FP64 reference. Keep the scenario matrix in pytest and use seeded bounded lengths while preserving rank and document boundary cases. Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Remove the TP+CP-only case and retain worker output only for failures. Document token slicing and simplify scenario dispatch. Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
5af3bc6 to
d4d62a8
Compare
|
/te-ci pytorch L0 L1 |
[Medium impact] The changes affect the Gated DeltaNet distributed-attention workflow. The PR appears safe to merge, with a non-blocking input check recommended for dense sequence offsets. Findings
SummaryAdds contiguous context parallelism to Gated DeltaNet attention.
One non-blocking hardening issue remains: dense sequence offsets should be checked against row boundaries. DiagramsequenceDiagram
participant R as CP ranks
participant C as CUDA state composition
participant A as Local GDN attention
R->>R: Gather forward summaries
R->>C: Compose starting states
C->>A: Supply local starting states
A-->>R: Local outputs and ending states
R->>R: Gather backward summaries
R->>C: Compose ending-state gradients
C->>A: Supply gradients for local backward
A-->>R: Local input gradients
Reviews (1) · Last reviewed commit: "Rename GDN CP length generator" · Reviewed by Greptile |
| if cu_seqlens.numel() != batch + 1: | ||
| raise ValueError("Dense cu_seqlens must have shape [batch_size + 1].") | ||
| return cu_seqlens, length |
There was a problem hiding this comment.
Dense offsets can mix documents
The new dense path checks the size of cu_seqlens, but not whether its offsets match the batch rows. For example, a [2, 128, H, D] input accepts [0, 64, 256]. The packed cuDNN call then resets state halfway through the first row and carries the next document across the row boundary.
Reject offsets other than [0, S, 2*S, ..., B*S], as the size-one CP path already does. This would catch accidental document mixing with a clear input error.
Description
Add context parallelism (CP) to PyTorch GatedDeltaNetAttention, allowing long sequences and packed documents to be processed across multiple GPUs.
Each rank owns a contiguous portion of the input. TE uses cuDNN Frontend to compute local state summaries, exchanges summaries across ranks, and calculates the starting memory needed for local attention. Backward propagates state gradients across the same document boundaries.
This PR enables GDN with contiguous ownership. The shared metadata and state-composition interfaces are designed for reuse by other linear attention variants.
Type of change
Changes
Validation
Eight distributed scenarios passed at both CP=2 and CP=4 on GB200, along with 25 C++ state-composition tests. Changed-file formatting and lint checks passed; full QA Python lint encountered existing upstream tooling issues.
Checklist: