Skip to content

[PyTorch] Drop gradient of zero-prob rows in moe_permute_with_probs backward - #3901

Open
ArchanaChetan07 wants to merge 3 commits into
NVIDIA:mainfrom
ArchanaChetan07:fix/permute-with-probs-zero-prob-grad
Open

ArchanaChetan07 wants to merge 3 commits into
NVIDIA:mainfrom
ArchanaChetan07:fix/permute-with-probs-zero-prob-grad

Conversation

@ArchanaChetan07

Copy link
Copy Markdown

Description

moe_permute_with_probs / moe_permute_and_pad_with_probs (mask map) write a zero row into the permuted output when a routed entry's prob is exactly 0.0. This is how routing-map padding is represented (_permute_kernel, if prob == 0.0). That output row does not depend on the input, but the backward (te_moe::permute_mask_map_bwd → unpermute_with_mask_map) accumulates every routed row of the incoming gradient. So whatever gradient a downstream op puts on a padded row leaks into the input gradient.

Repro from #3887 (routing [[1,1],[0,1]], probs [[1,0],[0,1]], row weights 1/10/100):

input grad of token 0
main [11, 11] ❌
this PR [1, 1] ✅ (matches PyTorch autograd of stack([x0, 0, x1]))

Fixes #3887

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • _moe_permute_mask_map_setup_context also saves permuted_probs when probs was given (no extra state when it wasn't).
  • _moe_permute_mask_map_backward_wrapper zeros the gradient rows where permuted_probs == 0 before calling the unchanged backward op. The prob gradient path is untouched.
  • New test test_permutation_mask_map_with_zero_probs_backward (fp32/fp16, eager and torch.compile(fullgraph=True)). It zeros about 1/3 of the routed probs and checks output, permuted probs, and input grad bit-exactly against the existing pytorch_permute_mask_map reference, using small-integer data.

Design note: I masked in the PyTorch frontend instead of the Triton kernel. Masking in the kernel would mean adding a forward-probs pointer to _unpermute_kernel, whose permuted_probs_ptr slot is already used for the prob gradient in this backward. It would also change the kernel signature shared with JAX. The cost is one extra elementwise pass over grad_output, and only when probs are given. If you'd prefer the in-kernel version, I'm happy to rework it. I haven't checked whether the JAX permutation backward has the same issue. Its zero-prob forward branch lives in the same kernel, so it may be worth a follow-up.

Testing

Run on a local Turing GPU (T1000, sm_75), torch 2.6.0+cu124, triton 3.2. I loaded TE's real transformer_engine/pytorch/permutation.py and the Triton kernels, with stubs only for the compiled transformer_engine_torch extension, which this path doesn't use:

  • The issue repro: fails on main, passes with this PR.
  • The new test over 5 seeds × {fp32, fp16} × {eager, compile}: all pass. With the fix reverted, it fails (input grad mismatch).

I have not run the full tests/pytorch/test_permutation.py suite or the L0 launchers on a Hopper/Blackwell machine. CI coverage would be appreciated.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

🤖 Generated with Claude Code

…ackward

The mask-map permute kernel writes a zero row (instead of the token)
whenever the routed prob is exactly zero, which is how routing-map
padding is represented. The backward, however, unpermutes every routed
row of the incoming gradient, so a nonzero gradient on one of those
zero rows leaked into the input gradient even though the forward output
did not depend on the input there.

Save the permuted probs when probs are given and zero the matching
gradient rows before unpermuting. Add a regression test that compares
against a PyTorch reference with exact integer data.

Fixes NVIDIA#3887

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ArchanaChetan07 <apatil@sandiego.edu>
@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Nov 19, 2026
@ArchanaChetan07
ArchanaChetan07 marked this pull request as ready for review November 19, 2026 02:48
@greptile-apps

greptile-apps Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking issues were found.

Summary

The PR drops input gradients for rows whose routing probability is zero, while keeping probability gradients unchanged.

  • Zero-probability routed rows no longer add to input gradients.

Reviews (4) · Last reviewed commit: "[PyTorch] Cover bf16 and non-finite padd..." · Reviewed by Greptile

Comment thread tests/pytorch/test_permutation.py Resolved
Zero-prob entries are still copied to the permuted probs, so they must
keep receiving a prob gradient even though their activation gradient is
dropped. Make probs require grad, feed a nonzero gradient for the
permuted probs, and compare probs.grad against the reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ArchanaChetan07 <apatil@sandiego.edu>
Masking the gradient of zero-prob rows must be a select, not a multiply:
an inf/NaN gradient on a padded row would otherwise turn into NaN
(inf * 0) and leak into the input gradient. Parametrize the regression
test over finite/inf/nan gradients on the padded rows, build the
reference with torch.where for the same reason, and add bf16 when the
GPU supports it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ArchanaChetan07 <apatil@sandiego.edu>
@ArchanaChetan07

Copy link
Copy Markdown
Author

Since #3893 was closed in favor of this PR, I've folded in the two points from it and pushed bafa805:

  • Select, not multiply. The fix already used masked_fill, so a non-finite gradient on a padded row can't become inf * 0 = NaN. The regression test now checks this directly: it's parametrized over finite / inf / nan gradients on the zero-prob rows, and the reference uses torch.where for the same reason. A multiply-based mask fails every inf / nan case.
  • No probs. permuted_probs is saved as None when probs isn't given, so masking is skipped. The existing no-probs and empty-input mask-map tests cover that path.

The test also checks the prob gradient now (2295d58): zero-prob entries must still receive their prob gradient. bf16 is added, gated on te.is_bf16_available().

Local validation (T1000 / sm_75, torch 2.6.0+cu124, triton 3.2). The compiled TE extension isn't available on this machine, so I ran the real transformer_engine/pytorch/permutation.py and Triton kernels with the extension stubbed out:

  • New test, fp32/fp16 × finite/inf/nan × eager/torch.compile × 3 seeds: 36/36 pass. On main, every case fails.
  • Existing mask-map and chunk tests in test_permutation.py: 100 passed. The only failures don't depend on this change:
    • 2 bf16 capacity_drop cases fail because Triton can't build bf16 for sm_75.
    • The padding tests can't run here, since Fp8Padding needs the compiled extension. They fail identically on main.
  • bf16, the padding tests, and the rest of the suite still need CI on a supported GPU.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[PyTorch] moe_permute_with_probs: wrong input gradient for a routed zero-probability entry

1 participant