Skip to content

[JAX] MoEBlock aware of Dense TP axes (compound EP) - #3894

Open
tdophung wants to merge 15 commits into
NVIDIA:mainfrom
tdophung:jax-moe-compound-ep
Open

tdophung wants to merge 15 commits into
NVIDIA:mainfrom
tdophung:jax-moe-compound-ep

Conversation

@tdophung

@tdophung tdophung commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

The TE MoEBlock must know when the expert-parallel (EP) group spans several mesh axes, for example EP over ("expert", "tensor") when tensor parallelism is folded into EP for the MoE region (EP+TP+FSDP layouts). This PR lets MeshResource.ep_resource name a tuple of mesh axes and makes the MoE path and the EP bootstrap resolve EP, DP and FSDP the same way from one MeshResource.

It also fixes the EP-group count on meshes with both DP and FSDP: the bootstrap now counts groups over every non-EP outer axis, which a dp x fsdp x ep mesh needs.

This PR supersedes #3462. It carries that PR's net changes by Jeremy Berchtold, rebased on current main (which now includes #3883), followed by follow-up fixes. Four of #3462's commits that cancel each other out (a change and its revert) are not replayed.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • MeshResource.ep_resource may be a tuple of mesh axes (compound EP); the MoEBlock and moe() handle compound EP, including with FSDP and with rank-2 inputs.
  • EP roles are named only through MeshResource: ep_bootstrap takes mesh_resource= (explicit, else the global resource), and the standalone EP ops resolve EP from it.
  • EP-group counting and the bind-time EP axes flatten tuple FSDP resources and drop EP axes, reusing _fsdp_outer_axes from [JAX] Control MeshResource at API call granularity for Dense APIs #3883; the group count covers every non-EP outer axis.
  • A size-one member of a compound EP axis is accepted in the EP leading-axis check; the default EP output partition spec (flattened in [JAX] Control MeshResource at API call granularity for Dense APIs #3883) is tested with tuple FSDP and EP resources.
  • ep_bootstrap rejects a mesh_resource that is not a MeshResource.
  • Tests: compound EP and EP+TP layouts in the TE EP MoE suite, a second pass on a dp x fsdp x ep mesh (dp=2 only with 8 or more GPUs), and unit tests for the axis helpers.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Co-authored-by: Jeremy Berchtold jberchtold@nvidia.com

@tdophung
tdophung force-pushed the jax-moe-compound-ep branch from bb2a836 to 9d118d1 Compare November 19, 2026 04:54
@tdophung

Copy link
Copy Markdown
Collaborator Author

/te-ci jax

1 similar comment
@tdophung

Copy link
Copy Markdown
Collaborator Author

/te-ci jax

@tdophung
tdophung force-pushed the jax-moe-compound-ep branch 2 times, most recently from 7eafb63 to d86af65 Compare November 19, 2026 23:39
@tdophung

Copy link
Copy Markdown
Collaborator Author

/te-ci L1 jax

1 similar comment
@tdophung

Copy link
Copy Markdown
Collaborator Author

/te-ci L1 jax

jberchtold-nvidia and others added 13 commits November 19, 2026 18:23
Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com>
Signed-off-by: Teddy Do <tdophung@nvidia.com>
Signed-off-by: Teddy Do <tdophung@nvidia.com>
Signed-off-by: Teddy Do <tdophung@nvidia.com>
Signed-off-by: Teddy Do <tdophung@nvidia.com>
Signed-off-by: Teddy Do <tdophung@nvidia.com>
Signed-off-by: Teddy Do <tdophung@nvidia.com>
Signed-off-by: Teddy Do <tdophung@nvidia.com>
@tdophung
tdophung force-pushed the jax-moe-compound-ep branch from d86af65 to 7501b01 Compare November 20, 2026 02:23
@tdophung

Copy link
Copy Markdown
Collaborator Author

/te-ci L1 jax

@tdophung
tdophung marked this pull request as ready for review November 20, 2026 02:28
@greptile-apps

greptile-apps Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issues were found.

Summary

Adds compound EP axes through MeshResource, supports rank-2 MoE inputs, and counts both DP and FSDP outer groups.

  • MoE blocks can treat expert and tensor axes as one EP group.
  • EP setup and output layouts now count every outer DP and FSDP axis.
  • MoE accepts flat token inputs and keeps their original shape.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  R[MeshResource] --> B[EP bootstrap]
  R --> M[MoE input layout]
  R --> O[Default EP output layout]
  B --> G[Compound EP groups]
  M --> D[Dispatch and expert work]
  G --> D
  D --> C[Combine and backward]
  O --> C
Loading

Reviews (3) · Last reviewed commit: "Check the standalone EP round trip witho..." · Reviewed by Greptile

Comment thread transformer_engine/jax/ep.py Resolved
Signed-off-by: Teddy Do <tdophung@nvidia.com>
@tdophung

Copy link
Copy Markdown
Collaborator Author

/te-ci L1 jax

Signed-off-by: Teddy Do <tdophung@nvidia.com>
@tdophung

Copy link
Copy Markdown
Collaborator Author

/te-ci L1 jax

@tdophung

Copy link
Copy Markdown
Collaborator Author

/te-ci jax

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants