Repository navigation
Conversation
[Medium impact] The changes reach supported PyTorch expert-parallel MoE workflows and their shared EP communication setup. Fix the SM 10.7 workspace query before merging so supported MXFP8 training runs do not fail in backward. FindingsSummaryAdds expert-parallel MoE overlap for BF16 and MXFP8.
Diagram%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[Input tokens] --> B[Dispatch]
B -->|Ready rows| C[FC1 and SwiGLU]
C --> D[FC2]
D -->|Ready columns| E[Combine]
E --> F[Output tokens]
Reviews (3) · Last reviewed commit: "[Common][PyTorch] Allocate the overlappe..." · Reviewed by Greptile |
| config = self._overlap_config() | ||
| ops = self._moe_ops(config, self._make_buffer_from_config(config)) | ||
| ref_ops = self._moe_ops(config, self._make_buffer_from_config(config)) |
There was a problem hiding this comment.
test_moe_overlap_cuda_graph builds both models with _overlap_config() and never enables MXFP8 autocast. It therefore misses capture and replay of the new shared dispatch window, requantization, and MXFP8 weight gradients. The tests could pass while that training path breaks.
Run this test with both BF16 and MXFP8BlockScaling, using MXFP8 transport for the latter. Compare replayed outputs and gradients after changing the inputs and routing.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
82d9201 to
7c92f4a
Compare
…) with in-call weight gradients for BF16 and MXFP8 Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
… MoE EP GEMMs Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
… and tidy the EP C API and CuTeDSL wrappers Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
7c92f4a to
6e27c25
Compare
| for (const bool fp32_out : {false, true}) { | ||
| const ep::EpWgradConfig config{rows, m, n, num_experts, 1, fp32_out, | ||
| false, false, 0, mxfp8, false, ctx.alignment(), | ||
| routed_rows}; | ||
| const auto layout = config.workspace_layout(); |
There was a problem hiding this comment.
Supported MXFP8 backward calls fail
On SM 10.7 with MXFP8 and 256 receive rows, wgrad_workspace_bytes queries both BF16 and FP32 output types. The FP32 query always raises because build_kernel rejects that capacity, even when the actual BF16 weight gradient is supported. The Python check allows those BF16 shapes, so the run selects FusedMoE, completes forward, and fails while sizing the backward workspace.
Size the workspace from supported variants, or pass the actual output type into the query.
… GEMM configs and reject unsupported recipes Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
…P transport recipe Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
… paged stashing Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
…e message Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
…ing cuDNN upstreaming Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
Description
Please include a brief summary of the changes, relevant motivation and context.
Fixes # (issue)
Type of change
Changes
Please list the changes introduced in this PR:
Checklist: