Repository navigation
[PyTorch] Drop gradient of zero-prob rows in moe_permute_with_probs backward - #3901
ArchanaChetan07 wants to merge 3 commits into
Conversation
…ackward The mask-map permute kernel writes a zero row (instead of the token) whenever the routed prob is exactly zero, which is how routing-map padding is represented. The backward, however, unpermutes every routed row of the incoming gradient, so a nonzero gradient on one of those zero rows leaked into the input gradient even though the forward output did not depend on the input there. Save the permuted probs when probs are given and zero the matching gradient rows before unpermuting. Add a regression test that compares against a PyTorch reference with exact integer data. Fixes NVIDIA#3887 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ArchanaChetan07 <apatil@sandiego.edu>
[Medium impact] The PR appears safe to merge; no blocking issues were found. SummaryThe PR drops input gradients for rows whose routing probability is zero, while keeping probability gradients unchanged.
Reviews (4) · Last reviewed commit: "[PyTorch] Cover bf16 and non-finite padd..." · Reviewed by Greptile |
Zero-prob entries are still copied to the permuted probs, so they must keep receiving a prob gradient even though their activation gradient is dropped. Make probs require grad, feed a nonzero gradient for the permuted probs, and compare probs.grad against the reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ArchanaChetan07 <apatil@sandiego.edu>
Masking the gradient of zero-prob rows must be a select, not a multiply: an inf/NaN gradient on a padded row would otherwise turn into NaN (inf * 0) and leak into the input gradient. Parametrize the regression test over finite/inf/nan gradients on the padded rows, build the reference with torch.where for the same reason, and add bf16 when the GPU supports it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ArchanaChetan07 <apatil@sandiego.edu>
|
Since #3893 was closed in favor of this PR, I've folded in the two points from it and pushed
The test also checks the prob gradient now ( Local validation (T1000 / sm_75, torch 2.6.0+cu124, triton 3.2). The compiled TE extension isn't available on this machine, so I ran the real
|
Description
moe_permute_with_probs/moe_permute_and_pad_with_probs(mask map) write a zero row into the permuted output when a routed entry's prob is exactly0.0. This is how routing-map padding is represented (_permute_kernel,if prob == 0.0). That output row does not depend on the input, but the backward (te_moe::permute_mask_map_bwd→unpermute_with_mask_map) accumulates every routed row of the incoming gradient. So whatever gradient a downstream op puts on a padded row leaks into the input gradient.Repro from #3887 (routing
[[1,1],[0,1]], probs[[1,0],[0,1]], row weights 1/10/100):main[11, 11]❌[1, 1]✅ (matches PyTorch autograd ofstack([x0, 0, x1]))Fixes #3887
Type of change
Changes
_moe_permute_mask_map_setup_contextalso savespermuted_probswhenprobswas given (no extra state when it wasn't)._moe_permute_mask_map_backward_wrapperzeros the gradient rows wherepermuted_probs == 0before calling the unchanged backward op. The prob gradient path is untouched.test_permutation_mask_map_with_zero_probs_backward(fp32/fp16, eager andtorch.compile(fullgraph=True)). It zeros about 1/3 of the routed probs and checks output, permuted probs, and input grad bit-exactly against the existingpytorch_permute_mask_mapreference, using small-integer data.Design note: I masked in the PyTorch frontend instead of the Triton kernel. Masking in the kernel would mean adding a forward-probs pointer to
_unpermute_kernel, whosepermuted_probs_ptrslot is already used for the prob gradient in this backward. It would also change the kernel signature shared with JAX. The cost is one extra elementwise pass overgrad_output, and only when probs are given. If you'd prefer the in-kernel version, I'm happy to rework it. I haven't checked whether the JAX permutation backward has the same issue. Its zero-prob forward branch lives in the same kernel, so it may be worth a follow-up.Testing
Run on a local Turing GPU (T1000, sm_75), torch 2.6.0+cu124, triton 3.2. I loaded TE's real
transformer_engine/pytorch/permutation.pyand the Triton kernels, with stubs only for the compiledtransformer_engine_torchextension, which this path doesn't use:main, passes with this PR.I have not run the full
tests/pytorch/test_permutation.pysuite or the L0 launchers on a Hopper/Blackwell machine. CI coverage would be appreciated.Checklist:
🤖 Generated with Claude Code