Skip to content

[PyTorch] Accumulate fused wgrads of an overwrite_main_grad weight used more than once per backward - #3821

Open
rapatel wants to merge 9 commits into
NVIDIA:mainfrom
rapatel:fix/overwrite-main-grad-reused-weight
Open

rapatel wants to merge 9 commits into
NVIDIA:mainfrom
rapatel:fix/overwrite-main-grad-reused-weight

Conversation

@rapatel

@rapatel rapatel commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Description

With fuse_wgrad_accumulation=True and Megatron-FSDP's overwrite_main_grad, a Linear or LayerNormLinear weight used twice in one backward (an LM head shared with an MTP head) kept only the last use's weight gradient; it now overwrites on the first wgrad and accumulates the rest.

Fixes # (issue)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Linear and LayerNormLinear run the fused wgrad GEMM with accumulate=False whenever the weight has overwrite_main_grad=True. Megatron-FSDP sets that flag so the first wgrad after a reduce-scatter overwrites the stale main_grad instead of needing a memset. A second use of the weight in the same backward then overwrites the first use's wgrad.
  • The fix moves the decision into the wgrad GEMM closure (overwrite_main_grad_wgrad_gemm in module/_common.py):
    • accumulate if the weight's _fsdp_wgrad_written is set, else overwrite
    • set _fsdp_wgrad_written after the GEMM.

Megatron-FSDP clears the flag after every backward, where it consumes or zeroes main_grad. That is a companion Megatron-LM PR (NVIDIA/Megatron-LM#8991). A weight that has no _fsdp_wgrad_written attribute keeps today's overwrite, so this PR is safe to land before that one and the bug is fixed once both have landed.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

…ed more than once per backward

Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
@rapatel
rapatel requested a review from ksivaman as a code owner November 5, 2026 02:54
@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Nov 5, 2026
@greptile-apps

greptile-apps Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no outstanding findings remain.

Summary

The PR lets reused weights overwrite main_grad once, then accumulate later weight gradients.

  • Fused gradients overwrite once, then add later uses of the same weight.
  • CUDA graph replays keep weight-gradient flags in step with captured work.

All five previous, unnumbered findings are addressed. No new issues were found in the latest change.

Reviews (8) · Last reviewed commit: "Clarify that capture-time hooks do not r..." · Reviewed by Greptile

…rad closures small, document the flag

Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
…hed microbatches sharing a weight

Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
Comment thread transformer_engine/pytorch/graph.py Resolved
… clearing them per backward

Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
@rapatel

rapatel commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 4e81d0e

@rapatel

rapatel commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

/te-ci pytorch

@ksivaman

Copy link
Copy Markdown
Member

/te-ci pytorch

Comment thread transformer_engine/pytorch/graph.py Resolved
@greptile-apps

greptile-apps Bot commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Want your agent to iterate on Greptile's feedback? Try greploops.

…mulate

Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
@rapatel

rapatel commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

/te-ci pytorch

@rapatel

rapatel commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

@ksivaman @ptrendx Gentle reminder if one of you are able to help approve this. Thanks!

@CodersAcademy006 CodersAcademy006 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read this against the tree at b8d5a61 and against the companion Megatron-LM#8991, since the contract only makes sense with both halves in view.

The land-order claim in the description holds, and the reason is worth stating. The TE GEMM helper never creates the attribute and never clears one: it reads _fsdp_wgrad_written with a None default and writes it back only if written is not None. #8991 owns the lifecycle from both ends. Its layers.py write is guarded by hasattr(weight, "__fsdp_param__"), while the clears are deliberately broader and cover three different moments: _root_pre_backward at backward entry, _grad_acc once a param's gradient has been handled, and zero_grad_buffer across trainable params at the start of an iteration. Between them the flag does not carry into the next backward's accumulation window. Against current Megatron-FSDP main the attribute is absent, so TE's None path keeps today's overwrite behavior until #8991 lands.

One lifecycle note. grad_added_to_main_grad looks like the obvious candidate for a fail-fast check on a missed clear, and it cannot serve: the unshard between two uses of the same weight resets it, which is the reason this separate flag exists at all, and #8991's own comment on the layers.py hunk says so. The two have different lifetimes by design, so any such check has to sit where the owner's consumption boundary is actually known.

Three places this is easy to get wrong, and all three are right here:

  • The flag is read when wgrad_gemm runs, so a closure queued into wgrad_store in backward order still picks overwrite or accumulate in GEMM order. The delay=True parameterisation covers it.
  • _snapshot_wgrad_written and _restore_wgrad_written put the flags back after warmup, so a capture bakes the choice the caller's flags implied. _newly_wgrad_written records params whose flag was falsey before capture and truthy after, and since the helper leaves the absent-attribute path untouched, a weight that was never in the protocol stays off the replay list.
  • _reset_main_grad fills main_grad with 1e3. Zeroing it would have made a wrongly baked accumulate=True numerically identical to an overwrite, and the test would pass either way.

One doc clarification left inline on the capture_time_hooks entry, where two descriptions of the same hook are not equally precise.

Comment thread transformer_engine/pytorch/graph.py Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants