Repository navigation
Conversation
[Medium impact] The PR appears safe to merge; no blocking issues were found. SummaryThe PR keeps specialized MXFP8 casts only when both scale pitches match their compiled values. Other pitches use the existing runtime-stride kernel.
Diagram%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Bidirectional MXFP8 cast] --> B{Both pitches match compiled values?}
B -->|Yes| C{2048 or 4096 columns?}
C -->|Yes| D[Specialized kernel]
C -->|No| E[Runtime-stride kernel]
B -->|No| E
Reviews (9) · Last reviewed commit: "Merge branch 'main' into bvolpato/fix-mx..." · Reviewed by Greptile |
|
/te-ci L0 |
|
Merged current main ( Rebuilt the common library and PyTorch extension on an RTX 5070 Ti (SM120), CUDA 13.2 and PyTorch 2.12.1, with CuTeDSL support enabled and NCCL EP disabled. The CUDA pitched-scale regression passes. Backend-routing probes also confirm that the fixture restores CuTeDSL dispatch after normal completion and an injected failure. Scoped pre-commit, C++ lint, and the license check pass. The previous runtime failures need CI-side follow-up:
Could the four |
|
Want your agent to iterate on Greptile's feedback? Try greploops. |
|
/te-ci |
|
Merged current Validation on the merged tree passed:
The GPU checks used one RTX 5070 Ti (SM120), driver 595.71.05, PyTorch 2.12.1, CUDA 13.2, cuDNN 9.20, and NCCL 2.29.7. Full L0 and A100/H100/B200 execution have not been rerun locally. The remaining upstream checks need CI-side follow-up. The new Documentation, License, Lint, and Minimum Supported Versions runs are The previous runtime logs identify separate dependency issues:
Could a maintainer approve the gated workflows and rerun L0 on |
|
/te-ci |
|
Sorry there was a CI issue which should be fixed by now. I'm re-triggering another CI |
|
Merged current Rebuilt the common library and PyTorch extension from the merged tree, with CuTeDSL support enabled and NCCL EP disabled. Verified that both imported libraries came from this checkout. Validation passed:
The grouped checks can be rerun with: python -m pytest tests/pytorch/test_fusible_ops.py::TestFuser::test_deferred_param_init -k grouped_linear -q
PYTORCH_JIT=0 NVTE_TORCH_COMPILE=0 python -m pytest tests/pytorch/test_ops_grouped_linear_distributed_weight.py -qLocal GPU validation used one RTX 5070 Ti (SM120), driver 595.71.05, PyTorch 2.12.1, CUDA 13.2, cuDNN 9.20, and NCCL 2.29.7. A100/H100/B200 L0 still needs fresh execution on this head. The four new public workflows are |
The 2048- and 4096-column bidirectional MXFP8 cast specializations hardcode Transformer Engine's default padded scale strides, so caller-provided scale pitches write exponents to the wrong offsets. Dispatch to those fixed-width specializations only when both rowwise and columnwise scale strides match the expected padded constants, and route custom pitches through the existing runtime-stride kernel. Add GPU regression coverage on the CUDA backend across custom and default pitches. Signed-off-by: Bruno Volpato <brunocvcunha@gmail.com>
5826a29 to
07d00eb
Compare
|
rebased onto current @kainzhong @Oleg-Goncharov could you re-approve and trigger the gated workflows + also heads-up for @kainzhong: I noticed |
|
/te-ci |
Description
Prevent silent scale corruption when bidirectional BF16-to-MXFP8 quantization writes to output buffers with caller-provided scale pitches. The FP8 data can be correct while misplaced scales cause downstream dequantization to use the wrong exponents.
The 2048- and 4096-column specializations bake in TE's default padded scale strides. Use them only when both supplied strides match those constants. Custom layouts use the existing runtime-stride kernel, while default layouts retain their specialized dispatch.
Related to #3710.
Type of change
Changes
Before and after
A CUDA E4M3 reproducer on an RTX 5070 Ti used seed 1234 and BF16 input
(8 * torch.randn((64, 2048), device="cuda")).bfloat16(). The unswizzled output scale shapes were(64, 68)and(2, 2064), adding 4 rowwise and 16 columnwise pitch entries.64 x 64scale bytes2 x 2048scale bytesThe FP8 payload matched exactly in both runs. The fix eliminated the scale mismatches, and the scale-buffer guards remained untouched.
Validation
The before/after comparison used CUDA 13.2 and PyTorch 2.12.1 on SM120, with the CUDA quantization backend selected. The focused regression fails before the fix and passes afterward.
Rebuilt the common library and PyTorch extension for the merged tree at
dcc0aa28, with CuTeDSL support enabled and NCCL EP disabled. Verified that the imported libraries came from the tested checkout. The pitched-scale regression and backend-routing probes passed, including restoration after an injected failure. Direct CUDA launcher checks, Compute Sanitizer memcheck, scoped pre-commit, C++ lint, and the license check also passed.An earlier merge head (
94199c08) with the same two-file diff was also compared with its base (63451d6b) on NVIDIA B200 (SM100, driver 580.95.05), before rebasing and squashing onto currentmain(07d00eb1). Both trees were built for100awith CuTeDSL support enabled and NCCL EP disabled, and imports resolved to each tree. Two toolchains were used: PyTorch 2.13.0+cu130 with CUDA 13.0, cuDNN 9.20, and cuBLAS 13.4, and PyTorch 2.15.0+cu132 with CUDA 13.2, cuDNN 9.26, cuBLAS 13.4, and FlashAttention 2.8.4.NVTE_ENABLE_CUTEDSL_BACKENDset to0and1.K=2048andK=4096instantiations for default layouts andK=0for custom pitches.NVIDIA_TF32_OVERRIDE=0. The failures on both trees aretest_torch_compile.py([PyTorch] torch.compile fails on FusedAttnBackend IntEnum comparisons #3716) andtest_dot_product_attention_score_modintest_flex_attention.py. The second one comes fromnvidia-cudnn-frontend1.31.0 and newer, which this run installed, and [PyTorch] Keep Flex Attention score_mod graphs on the cuDNN backend #3796 fixes it.main(b0219803, which includes [Common][PyTorch] Add strided batched GEMM in BF16/MXFP8 #3160) and rebuilt with the second toolchain. The regression and the separate decode check fail without the kernel change and find no failures with it, default layouts keep their specialized kernels, and no L0 PyTorch test failed on only one tree.tests/pytorch/mxfp8with the CuTeDSL backend enabled found no failures on either tree. That backend rejects these compact scale shapes withInvalid mS_row.shape[0]on both trees, which is one more reason the regression stays on CUDA.No program under
benchmarks/covers this cast, so a one-off script measured it on one B200 with the second toolchain, the CUDA backend, and BF16 input from8 * randnto E4M3 with unswizzled scales. Each run used 50 warm-up calls, then 400 timed calls withcuda.synchronize()after each. Kernel time comes fromtorch.profilerover 40 more calls. The values are medians over four alternating runs per build.K=2048/K=2048K=4096/K=4096K=2048/K=0K=4096/K=0For default layouts, the median time per call differs by at most about 1%, which is less than the spread between runs. For custom pitches, the base times come from output with wrong scales, so the difference is the cost of the correct kernel and not a regression.
Rerun the focused check with:
A100 and H100 were not run for this comparison. Upstream L0 still needs a fresh run on this head.
Checklist: