Repository navigation
Conversation
…hape is omitted Fixes NVIDIA#3888 Signed-off-by: lxcxjxhx <lxcxjxhx@users.noreply.github.com>
[Medium impact] The PR appears safe to merge; no actionable issues were found. SummaryFixes
No actionable issues found. Tests were not run. Reviews (1) · Last reviewed commit: "fix(unpermute): infer the token count fr..." · Reviewed by Greptile |
|
This one has never had a What it would exercise: the one-line change is in the I cannot run TransformerEngine here — no GPU and no built extension — so the local evidence is the CPU-only shape/logic path in that test. The |
Description
Fixes #3888.
moe_unpermute's mask-map branch falls back toinp.shapewhenrestore_shapeis omitted, butinpis the permuted buffer, so the output's firstdimension is short by the number of dropped routes. One line plus a regression test.
Type of change
Changes
transformer_engine/pytorch/permutation.py: derive the token count from the row mapinstead of the permuted buffer when
restore_shape is Nonetests/pytorch/test_permutation.py:test_unpermute_mask_map_dropped_row— two tokens,one with an all-zero routing row, asserting both the output shape and that the surviving
row lands in the right position
Checklist:
the row map, not
inp.shape, carries the token countnever had a
/te-cidispatch, and the box stays unticked until it does. The regression testwas verified on the CPU shape/logic path only (no GPU, no built extension here).
Original description (unchanged)
Fixes #3888. The launch shape is chosen in Python, so the fix is one line plus a regression test.
Mechanism
moe_unpermutemask-map branch (transformer_engine/pytorch/permutation.py:1104-1107):inpis the permuted buffer, soinp.shape[0]is the number of surviving routes. With top-1 routing a token whose routing row is all zeros is dropped, and the permuted buffer is one row shorter than the token count — so the first dimension of the output is wrong by exactly the number of dropped tokens, and the last surviving rows are cut off. Passingrestore_shapeexplicitly hides the bug, which is why the documented "for top-1 it can be omitted" path is the one that breaks.Change
if map_type == "mask": if restore_shape is None: - restore_shape = inp.shape + # The row count of `inp` is the number of surviving routes, which is + # smaller than the number of tokens once top-1 routing drops a token. + # The token count is the length of the row map. + restore_shape = (row_id_map.shape[0], inp.shape[1]) num_tokens, hidden_size = restore_shaperow_id_map.shape[0]is the token count by construction:moe_permuteallocates it as(num_tokens, routing_map.size(1) * 2 + 1)(permutation.py:295-297), and it checksinp.size(0) == routing_map.size(0)before permuting. Theindexbranch of the same function already derives its token count the same way (num_tokens = row_id_map.size(0),:1099), so this aligns the two paths instead of introducing a third convention.Test
test_unpermute_mask_map_infers_restore_shape_with_dropped_tokenintests/pytorch/test_permutation.py, using the reporter's four-token / three-route case: it checks the shape, the values against the input with the dropped token zeroed, and that the call with an explicitrestore_shapeagrees.I could not execute it — the Triton kernels need the compiled
transformer_engine_torchextension and a CUDA device, neither of which this machine has. The reporter reached the same failure throughTRITON_INTERPRET=1, which is the cheapest way to run the new test without a GPU if you want to confirm before CI; the assertion that fails without the fix is the shape check,(3, 3) != (4, 3).