Repository navigation
[PyTorch] Accumulate fused wgrads of an overwrite_main_grad weight used more than once per backward - #3821
[PyTorch] Accumulate fused wgrads of an overwrite_main_grad weight used more than once per backward#3821rapatel wants to merge 9 commits into
Conversation
…ed more than once per backward Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
for more information, see https://pre-commit.ci
[Medium impact] The PR appears safe to merge; no outstanding findings remain. SummaryThe PR lets reused weights overwrite
All five previous, unnumbered findings are addressed. No new issues were found in the latest change. Reviews (8) · Last reviewed commit: "Clarify that capture-time hooks do not r..." · Reviewed by Greptile |
…rad closures small, document the flag Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
…hed microbatches sharing a weight Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
… clearing them per backward Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
|
/ok to test 4e81d0e |
|
/te-ci pytorch |
|
/te-ci pytorch |
|
Want your agent to iterate on Greptile's feedback? Try greploops. |
…mulate Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
Signed-off-by: Ritesh Patel <ripatel@nvidia.com>
|
/te-ci pytorch |
CodersAcademy006
left a comment
There was a problem hiding this comment.
Read this against the tree at b8d5a61 and against the companion Megatron-LM#8991, since the contract only makes sense with both halves in view.
The land-order claim in the description holds, and the reason is worth stating. The TE GEMM helper never creates the attribute and never clears one: it reads _fsdp_wgrad_written with a None default and writes it back only if written is not None. #8991 owns the lifecycle from both ends. Its layers.py write is guarded by hasattr(weight, "__fsdp_param__"), while the clears are deliberately broader and cover three different moments: _root_pre_backward at backward entry, _grad_acc once a param's gradient has been handled, and zero_grad_buffer across trainable params at the start of an iteration. Between them the flag does not carry into the next backward's accumulation window. Against current Megatron-FSDP main the attribute is absent, so TE's None path keeps today's overwrite behavior until #8991 lands.
One lifecycle note. grad_added_to_main_grad looks like the obvious candidate for a fail-fast check on a missed clear, and it cannot serve: the unshard between two uses of the same weight resets it, which is the reason this separate flag exists at all, and #8991's own comment on the layers.py hunk says so. The two have different lifetimes by design, so any such check has to sit where the owner's consumption boundary is actually known.
Three places this is easy to get wrong, and all three are right here:
- The flag is read when
wgrad_gemmruns, so a closure queued intowgrad_storein backward order still picks overwrite or accumulate in GEMM order. Thedelay=Trueparameterisation covers it. _snapshot_wgrad_writtenand_restore_wgrad_writtenput the flags back after warmup, so a capture bakes the choice the caller's flags implied._newly_wgrad_writtenrecords params whose flag was falsey before capture and truthy after, and since the helper leaves the absent-attribute path untouched, a weight that was never in the protocol stays off the replay list._reset_main_gradfillsmain_gradwith1e3. Zeroing it would have made a wrongly bakedaccumulate=Truenumerically identical to an overwrite, and the test would pass either way.
One doc clarification left inline on the capture_time_hooks entry, where two descriptions of the same hook are not equally precise.
Description
With
fuse_wgrad_accumulation=Trueand Megatron-FSDP'soverwrite_main_grad, aLinearorLayerNormLinearweight used twice in one backward (an LM head shared with an MTP head) kept only the last use's weight gradient; it now overwrites on the first wgrad and accumulates the rest.Fixes # (issue)
Type of change
Changes
Please list the changes introduced in this PR:
LinearandLayerNormLinearrun the fused wgrad GEMM withaccumulate=Falsewhenever the weight hasoverwrite_main_grad=True. Megatron-FSDP sets that flag so the first wgrad after a reduce-scatter overwrites the stalemain_gradinstead of needing a memset. A second use of the weight in the same backward then overwrites the first use's wgrad.overwrite_main_grad_wgrad_gemminmodule/_common.py):_fsdp_wgrad_writtenis set, else overwrite_fsdp_wgrad_writtenafter the GEMM.Megatron-FSDP clears the flag after every backward, where it consumes or zeroes
main_grad. That is a companion Megatron-LM PR (NVIDIA/Megatron-LM#8991). A weight that has no_fsdp_wgrad_writtenattribute keeps today's overwrite, so this PR is safe to land before that one and the bug is fixed once both have landed.Checklist: