Skip to content

fix(permute): do not backpropagate through a zero-probability routed row - #3893

Closed
lxcxjxhx wants to merge 2 commits into
NVIDIA:mainfrom
lxcxjxhx:fix/permute-with-probs-zero-prob-gradient
Closed

lxcxjxhx wants to merge 2 commits into
NVIDIA:mainfrom
lxcxjxhx:fix/permute-with-probs-zero-prob-gradient

Conversation

@lxcxjxhx

Copy link
Copy Markdown
Contributor

Fixes #3887. The forward and the backward disagree about a zero-probability routed slot; the fix makes the backward differentiate what the forward actually computed.

Mechanism

The forward zeroes the row instead of copying the input (transformer_engine/common/triton/permutation.py:294-305):

        if PERMUTE_PROBS:
            prob = tl.load(probs_ptr + prob_off)
            ...
            if prob == 0.0:
                # for routing_map padding
                tl.store(output_ptr + output_off, 0.0, mask=mask)
            else:
                tl.store(output_ptr + output_off, inp, mask=mask)

so that row is a constant and d(out_row)/d(x) = 0. The backward wrapper (transformer_engine/pytorch/permutation.py:544-570) passed grad_output straight to unpermute_with_mask_map, which accumulates every routed row — so a gradient arriving for the padded row was added to the input, which is the [[11, 11], …] in the report.

Change

The permuted probabilities are already an output of the forward, so no kernel change and no new argument is needed — save them in the context and use them to zero the rows the forward zeroed:

-    ctx.save_for_backward(row_id_map, pad_offsets)
+    ctx.save_for_backward(row_id_map, pad_offsets, _permuted_probs)
...
-    row_id_map, pad_offsets = ctx.saved_tensors
+    row_id_map, pad_offsets, permuted_probs = ctx.saved_tensors
+
+    # The forward writes a zero row for a routed slot whose probability is zero (see the
+    # PERMUTE_PROBS branch of the kernel), so that row does not depend on the input and its
+    # incoming gradient must not reach it.
+    if permuted_probs.numel() > 0:
+        grad_output = grad_output * (permuted_probs != 0).unsqueeze(-1)

Notes on the shape of the change:

  • the mask is applied to the activation gradient only; the probability gradient travels on its own path (permuted_probs_grad → unpermute_with_mask_map), so it is untouched;
  • permuted_probs.numel() > 0 guards the moe_permute path without probabilities, where permuted_probs is an empty tensor and the kernel never zeroes rows — masking there would be wrong;
  • the multiplication also makes grad_output contiguous, which is what the custom op expects (the existing code did not call .contiguous()).

Test

test_permute_with_probs_zero_probability_row_has_no_input_gradient reproduces the report exactly: two tokens, token 0 routed twice with probabilities 1.0 and 0.0, row weights 1 / 10 / 100, then compares both the output and x.grad against torch.stack([x[0], zeros, x[1]]).

Not executed here — the kernels need the compiled extension and a CUDA device. Without the patch the assertion that fails is the gradient one ([[11, 11], [100, 100]] vs [[1, 1], [100, 100]]), and the reporter reproduced the same way through TRITON_INTERPRET=1 if you want to run it on CPU.

One thing worth a maintainer's eye before merging: this changes a gradient that existing tests may have pinned. I read the mask-map comparison tests and they compare against the same torch.stack-style reference, so they should agree with the fixed behaviour, but a test that was written against the previous accumulation would now fail — if that happens it is the old expectation that is wrong, not the fix.

@lxcxjxhx
lxcxjxhx requested a review from ksivaman as a code owner November 18, 2026 01:08
@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Nov 18, 2026
@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge based on the reviewed changes.

Summary

The PR stops gradients from zero-probability routed rows from reaching the input.

  • Zero-probability routed rows no longer send gradients to their input tokens.

Reviews (4) · Last reviewed commit: "[pre-commit.ci] auto fixes from pre-comm..." · Reviewed by Greptile

Signed-off-by: lxcxjxhx <lxcxjxhx@users.noreply.github.com>
@lxcxjxhx
lxcxjxhx force-pushed the fix/permute-with-probs-zero-prob-gradient branch from 6aa4bfb to e96fb3b Compare November 18, 2026 01:17
@lxcxjxhx

Copy link
Copy Markdown
Contributor Author

#3901 by @ArchanaChetan07 fixes this issue with a stronger test (dtype and torch.compile matrix, exact-equality assertions on both gradients), so I am closing this one rather than asking for two reviews of the same change.

Two details from my version that are worth carrying over if they are not already in #3901:

  1. Select, do not multiply, when masking the incoming gradient. grad_output * (permuted_probs != 0) is not equivalent to torch.where(...) in the presence of non-finite values: a padded row that arrives with an infinite gradient becomes inf * 0 = NaN, and the NaN then contaminates the accumulation for every other row. My branch used torch.where((permuted_probs != 0).unsqueeze(-1), grad_output, torch.zeros_like(grad_output)) for that reason.
  2. Guard the empty-probs case. moe_permute without probs also reaches the backward wrapper, where permuted_probs has numel() == 0; masking it against a non-empty grad_output would raise on the broadcast. Saving None (as [PyTorch] Drop gradient of zero-prob rows in moe_permute_with_probs backward #3901 does) covers it just as well.

Sorry for the duplicate; #3901 is the one to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[PyTorch] moe_permute_with_probs: wrong input gradient for a routed zero-probability entry

1 participant