Repository navigation
Conversation
…ne backward Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
Contributor
Author
|
/ok to test 82319b7 |
wujingyue
reviewed
Nov 4, 2026
wujingyue
left a comment
Contributor
There was a problem hiding this comment.
Thanks! Can you clean up the PR description?
wujingyue
approved these changes
Nov 4, 2026
… twice in one backward Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
Contributor
Author
|
/ok to test 8b9cbc2 |
…on-FSDP v1 only Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
Contributor
Author
|
/ok to test 4d9ae97 |
Contributor
|
/ok to test 368eddb |
1 task done
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
Contributor
|
/ok to test 290f057 |
Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
…ejected with Megatron-FSDP's default grad dtypes) Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
Contributor
Author
|
/ok to test 8cc95b2 |
8 of 13 tasks
Contributor
Author
|
/ok to test 333ebdd |
Contributor
Author
|
I believe codecov is at ~17% because it compares against the PR's old base. The files mentioned have nothing to do with these changes. |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
With gradient-accumulation fusion (GAF) under Megatron-FSDP v1, a weight used twice in one backward (e.g. the output layer shared by the LM head and the MTP head) kept only the last use's weight gradient. It now accumulates both.
Fix:
tensor_parallel/layers.py: accumulate when the new_fsdp_wgrad_writtenflag is set (oroverwrite_main_gradis False), and set the flag after the GEMM.megatron_fsdp.py: clear the flag wherevermain_gradis consumed or zeroed (post-backward grad processing, root pre-backward,zero_grad_buffer).The same bug in TE's fused wgrad (
te.LinearLM head, e.g. with MXFP8) is fixed by NVIDIA/TransformerEngine#3821, which reads the same flag. Each PR is safe to land on its own.@NVIDIA/mcore-oncall.Issue tracking
For PRs from open-source community contributors:
Linked issue:
Code provenance
You should always make a best effort to avoid copying code from other
repositories. In exceptional cases in which this is unavoidable and you copied
or adapted code from an external source, provide the following details and
copy to the file header:
Contribution process
Pre-checks
jit_fuser/torch.compile, CUDA extension, TE or external-library dispatch, or a scatter/index accumulation), I have added or updated its bit-exact determinism test and registered it intests/unit_tests/determinism/kernels/manifest.py(guide)Code review
Feel free to message or comment
@NVIDIA/mcore-oncallto help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.
Step 1: Mark PR as "Ready for Review"
.github/CODEOWNERS.Final Review might get declined if these requirements are not fulfilled.
Step 2: Final Review
For PRs that change
megatron/core, once all expert reviewers have approved, theFinal Reviewlabel is applied automatically and final reviewers are assigned.For PRs outside
megatron/core, this step is skipped.Step 3: Approved
Once all required reviewers have approved, the
Approvedlabel is applied automatically.Merge
Any member of mcore-engineers will be able to merge your PR.