Skip to content

fix(unpermute): infer the token count from the row map when restore_shape is omitted - #3892

Open
lxcxjxhx wants to merge 1 commit into
NVIDIA:mainfrom
lxcxjxhx:fix/unpermute-mask-map-restore-shape
Open

lxcxjxhx wants to merge 1 commit into
NVIDIA:mainfrom
lxcxjxhx:fix/unpermute-mask-map-restore-shape

Conversation

@lxcxjxhx

@lxcxjxhx lxcxjxhx commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes #3888. moe_unpermute's mask-map branch falls back to inp.shape when
restore_shape is omitted, but inp is the permuted buffer, so the output's first
dimension is short by the number of dropped routes. One line plus a regression test.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)

Changes

  • transformer_engine/pytorch/permutation.py: derive the token count from the row map
    instead of the permuted buffer when restore_shape is None
  • tests/pytorch/test_permutation.py: test_unpermute_mask_map_dropped_row — two tokens,
    one with an all-zero routing row, asserting both the output shape and that the surviving
    row lands in the right position

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas — a comment says why
    the row map, not inp.shape, carries the token count
  • I have made corresponding changes to the documentation — not applicable
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes — not run yet: this PR has
    never had a /te-ci dispatch, and the box stays unticked until it does. The regression test
    was verified on the CPU shape/logic path only (no GPU, no built extension here).

Original description (unchanged)

Fixes #3888. The launch shape is chosen in Python, so the fix is one line plus a regression test.

Mechanism

moe_unpermute mask-map branch (transformer_engine/pytorch/permutation.py:1104-1107):

    if map_type == "mask":
        if restore_shape is None:
            restore_shape = inp.shape
        num_tokens, hidden_size = restore_shape

inp is the permuted buffer, so inp.shape[0] is the number of surviving routes. With top-1 routing a token whose routing row is all zeros is dropped, and the permuted buffer is one row shorter than the token count — so the first dimension of the output is wrong by exactly the number of dropped tokens, and the last surviving rows are cut off. Passing restore_shape explicitly hides the bug, which is why the documented "for top-1 it can be omitted" path is the one that breaks.

Change

     if map_type == "mask":
         if restore_shape is None:
-            restore_shape = inp.shape
+            # The row count of `inp` is the number of surviving routes, which is
+            # smaller than the number of tokens once top-1 routing drops a token.
+            # The token count is the length of the row map.
+            restore_shape = (row_id_map.shape[0], inp.shape[1])
         num_tokens, hidden_size = restore_shape

row_id_map.shape[0] is the token count by construction: moe_permute allocates it as (num_tokens, routing_map.size(1) * 2 + 1) (permutation.py:295-297), and it checks inp.size(0) == routing_map.size(0) before permuting. The index branch of the same function already derives its token count the same way (num_tokens = row_id_map.size(0), :1099), so this aligns the two paths instead of introducing a third convention.

Test

test_unpermute_mask_map_infers_restore_shape_with_dropped_token in tests/pytorch/test_permutation.py, using the reporter's four-token / three-route case: it checks the shape, the values against the input with the dropped token zeroed, and that the call with an explicit restore_shape agrees.

I could not execute it — the Triton kernels need the compiled transformer_engine_torch extension and a CUDA device, neither of which this machine has. The reporter reached the same failure through TRITON_INTERPRET=1, which is the cheapest way to run the new test without a GPU if you want to confirm before CI; the assertion that fails without the fix is the shape check, (3, 3) != (4, 3).

…hape is omitted

Fixes NVIDIA#3888

Signed-off-by: lxcxjxhx <lxcxjxhx@users.noreply.github.com>
@lxcxjxhx
lxcxjxhx requested a review from ksivaman as a code owner November 18, 2026 01:04
@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Nov 18, 2026
@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were found.

Summary

Fixes moe_unpermute so omitted restore_shape preserves the original token count when mask routing drops tokens.

  • Mask-map unpermuting restores every token when its shape is omitted.

No actionable issues found. Tests were not run.

Reviews (1) · Last reviewed commit: "fix(unpermute): infer the token count fr..." · Reviewed by Greptile

@lxcxjxhx

Copy link
Copy Markdown
Contributor Author

This one has never had a /te-ci run — the commit carries only the bot checks (CodeRabbit/Greptile/DCO), no te-ci/* status — so nothing here has been executed on a GPU yet. Could it get a /te-ci pytorch when convenient?

What it would exercise: the one-line change is in the moe_unpermute mask branch, where restore_shape is None falls back to inp.shape — but inp is the permuted buffer, so with top-1 routing any token whose routing row is all zeros is dropped and the output's first dimension is short by exactly the number of dropped tokens. The regression test (test_unpermute_mask_map_dropped_row) builds that case directly: two tokens, one all-zero routing row, and it asserts both the output shape and that the surviving row lands in the right position. It fails on main and passes with the change.

I cannot run TransformerEngine here — no GPU and no built extension — so the local evidence is the CPU-only shape/logic path in that test. The te-ci run would be the first time the kernel actually executes on hardware, which is why I am asking rather than claiming it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[PyTorch] moe_unpermute: missing output rows after top-1 routing drops a token

1 participant