Skip to content

fix(megatron-fsdp): accumulate grad-accum-fusion wgrads of a weight used twice in one backward - #8991

Open
rapatel wants to merge 8 commits into
NVIDIA:mainfrom
rapatel:fix/mfsdp-gaf-twice-used-weight
Open

rapatel wants to merge 8 commits into
NVIDIA:mainfrom
rapatel:fix/mfsdp-gaf-twice-used-weight

Conversation

@rapatel

@rapatel rapatel commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

With gradient-accumulation fusion (GAF) under Megatron-FSDP v1, a weight used twice in one backward (e.g. the output layer shared by the LM head and the MTP head) kept only the last use's weight gradient. It now accumulates both.

Fix:

  • tensor_parallel/layers.py: accumulate when the new _fsdp_wgrad_written flag is set (or overwrite_main_grad is False), and set the flag after the GEMM.
  • megatron_fsdp.py: clear the flag wherever main_grad is consumed or zeroed (post-backward grad processing, root pre-backward, zero_grad_buffer).

The same bug in TE's fused wgrad (te.Linear LM head, e.g. with MXFP8) is fixed by NVIDIA/TransformerEngine#3821, which reads the same flag. Each PR is safe to land on its own.

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact @NVIDIA/mcore-oncall.

Issue tracking

For PRs from open-source community contributors:

  • New features: a linked issue is required. Please open a feature request and reference it here before submitting the PR.
  • Small updates (bug fixes, minor improvements): a linked issue is recommended and will accelerate the PR review process.

Linked issue:

Code provenance

You should always make a best effort to avoid copying code from other
repositories. In exceptional cases in which this is unavoidable and you copied
or adapted code from an external source, provide the following details and
copy to the file header:

  • Source URL, revision, and original file(s):
  • Files in this PR containing the copied or adapted code:
  • Applicable license(s):
  • Required copyright notices and attribution, and where they are retained or added:

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • If this PR adds or changes a GPU kernel (Triton, jit_fuser/torch.compile, CUDA extension, TE or external-library dispatch, or a scatter/index accumulation), I have added or updated its bit-exact determinism test and registered it in tests/unit_tests/determinism/kernels/manifest.py (guide)
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment @NVIDIA/mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

  1. When your PR is ready, click Ready for Review.
  2. An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
    • Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

…ne backward

Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
@rapatel
rapatel requested review from a team as code owners November 4, 2026 20:00
@rapatel rapatel self-assigned this Nov 4, 2026
@rapatel
rapatel enabled auto-merge November 4, 2026 20:01
@rapatel

rapatel commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 82319b7

@wujingyue wujingyue left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Can you clean up the PR description?

Comment thread megatron/core/tensor_parallel/layers.py Resolved
@nemo-automation-bot nemo-automation-bot Bot added the Final Review PR is in the "final review" stage label Nov 4, 2026
… twice in one backward

Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
@rapatel

rapatel commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 8b9cbc2

…on-FSDP v1 only

Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
@rapatel

rapatel commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 4d9ae97

@balasaajay

Copy link
Copy Markdown
Contributor

/ok to test 368eddb

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
@balasaajay

Copy link
Copy Markdown
Contributor

/ok to test 290f057

Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
…ejected with Megatron-FSDP's default grad dtypes)

Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
@rapatel

rapatel commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 8cc95b2

@rapatel

rapatel commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 333ebdd

@rapatel

rapatel commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

I believe codecov is at ~17% because it compares against the PR's old base. The files mentioned have nothing to do with these changes.

This branch was previously deployed

2 inactive deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Final Review PR is in the "final review" stage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants