Skip to content

[Common][PyTorch] EP Gemm Overlap - #3906

Draft
phu0ngng wants to merge 10 commits into
NVIDIA:mainfrom
phu0ngng:feat/moe-overlap-ops-main
Draft

phu0ngng wants to merge 10 commits into
NVIDIA:mainfrom
phu0ngng:feat/moe-overlap-ops-main

Conversation

@phu0ngng

Copy link
Copy Markdown
Collaborator

Description

Please include a brief summary of the changes, relevant motivation and context.

Fixes # (issue)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Change A
  • Change B

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@greptile-apps

greptile-apps Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The changes reach supported PyTorch expert-parallel MoE workflows and their shared EP communication setup.

Fix the SM 10.7 workspace query before merging so supported MXFP8 training runs do not fail in backward.

Findings

  1. P1 Supported MXFP8 backward calls fail ▶
  2. P2 MXFP8 replay lacks coverage ▶

Summary

Adds expert-parallel MoE overlap for BF16 and MXFP8.

  • Supported MoE operation sequences can run through one fused path.
  • Expert GEMMs can start work as EP data becomes ready.
  • EP calls can signal ready work and use separate SM budgets.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Input tokens] --> B[Dispatch]
  B -->|Ready rows| C[FC1 and SwiGLU]
  C --> D[FC2]
  D -->|Ready columns| E[Combine]
  E --> F[Output tokens]
Loading

Reviews (3) · Last reviewed commit: "[Common][PyTorch] Allocate the overlappe..." · Reviewed by Greptile

Comment on lines +1323 to +1325
config = self._overlap_config()
ops = self._moe_ops(config, self._make_buffer_from_config(config))
ref_ops = self._moe_ops(config, self._make_buffer_from_config(config))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 MXFP8 replay lacks coverage

test_moe_overlap_cuda_graph builds both models with _overlap_config() and never enables MXFP8 autocast. It therefore misses capture and replay of the new shared dispatch window, requantization, and MXFP8 weight gradients. The tests could pass while that training path breaks.

Run this test with both BF16 and MXFP8BlockScaling, using MXFP8 transport for the latter. Compare replayed outputs and gradients after changing the inputs and routing.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@phu0ngng
phu0ngng force-pushed the feat/moe-overlap-ops-main branch from 82d9201 to 7c92f4a Compare November 20, 2026 17:15
@phu0ngng phu0ngng added the 3.0 label Nov 20, 2026
…) with in-call weight gradients for BF16 and MXFP8

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
… MoE EP GEMMs

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
… and tidy the EP C API and CuTeDSL wrappers

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
@phu0ngng
phu0ngng force-pushed the feat/moe-overlap-ops-main branch from 7c92f4a to 6e27c25 Compare November 21, 2026 00:20
Comment on lines +527 to +531
for (const bool fp32_out : {false, true}) {
const ep::EpWgradConfig config{rows, m, n, num_experts, 1, fp32_out,
false, false, 0, mxfp8, false, ctx.alignment(),
routed_rows};
const auto layout = config.workspace_layout();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Supported MXFP8 backward calls fail

On SM 10.7 with MXFP8 and 256 receive rows, wgrad_workspace_bytes queries both BF16 and FP32 output types. The FP32 query always raises because build_kernel rejects that capacity, even when the actual BF16 weight gradient is supported. The Python check allows those BF16 shapes, so the run selects FusedMoE, completes forward, and fails while sizing the backward workspace.

Size the workspace from supported variants, or pass the actual output type into the query.

… GEMM configs and reject unsupported recipes

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
@phu0ngng
phu0ngng marked this pull request as draft November 21, 2026 03:38
Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
…P transport recipe

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
… paged stashing

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
…e message

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
…ing cuDNN upstreaming

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant