Repository navigation
Conversation
Contributor
[Medium impact] The PR appears safe to merge based on the reviewed changes. SummaryThe PR stops gradients from zero-probability routed rows from reaching the input.
Reviews (4) · Last reviewed commit: "[pre-commit.ci] auto fixes from pre-comm..." · Reviewed by Greptile |
Signed-off-by: lxcxjxhx <lxcxjxhx@users.noreply.github.com>
lxcxjxhx
force-pushed
the
fix/permute-with-probs-zero-prob-gradient
branch
from
November 18, 2026 01:17
6aa4bfb to
e96fb3b
Compare
for more information, see https://pre-commit.ci
Draft
8 of 13 tasks
Contributor
Author
|
#3901 by @ArchanaChetan07 fixes this issue with a stronger test (dtype and Two details from my version that are worth carrying over if they are not already in #3901:
Sorry for the duplicate; #3901 is the one to review. |
6 of 13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3887. The forward and the backward disagree about a zero-probability routed slot; the fix makes the backward differentiate what the forward actually computed.
Mechanism
The forward zeroes the row instead of copying the input (
transformer_engine/common/triton/permutation.py:294-305):so that row is a constant and
d(out_row)/d(x) = 0. The backward wrapper (transformer_engine/pytorch/permutation.py:544-570) passedgrad_outputstraight tounpermute_with_mask_map, which accumulates every routed row — so a gradient arriving for the padded row was added to the input, which is the[[11, 11], …]in the report.Change
The permuted probabilities are already an output of the forward, so no kernel change and no new argument is needed — save them in the context and use them to zero the rows the forward zeroed:
Notes on the shape of the change:
permuted_probs_grad→unpermute_with_mask_map), so it is untouched;permuted_probs.numel() > 0guards themoe_permutepath without probabilities, wherepermuted_probsis an empty tensor and the kernel never zeroes rows — masking there would be wrong;grad_outputcontiguous, which is what the custom op expects (the existing code did not call.contiguous()).Test
test_permute_with_probs_zero_probability_row_has_no_input_gradientreproduces the report exactly: two tokens, token 0 routed twice with probabilities1.0and0.0, row weights1 / 10 / 100, then compares both the output andx.gradagainsttorch.stack([x[0], zeros, x[1]]).Not executed here — the kernels need the compiled extension and a CUDA device. Without the patch the assertion that fails is the gradient one (
[[11, 11], [100, 100]]vs[[1, 1], [100, 100]]), and the reporter reproduced the same way throughTRITON_INTERPRET=1if you want to run it on CPU.One thing worth a maintainer's eye before merging: this changes a gradient that existing tests may have pinned. I read the mask-map comparison tests and they compare against the same
torch.stack-style reference, so they should agree with the fixed behaviour, but a test that was written against the previous accumulation would now fail — if that happens it is the old expectation that is wrong, not the fix.