Repository navigation
[Common] Use FP8 fast accumulation in the cuBLASMp comm+GEMM overlap - #3719
almogsegal wants to merge 1 commit into
Conversation
The cuBLASMp comm+GEMM backend ignored use_split_accumulator: nvte_all_gather_gemm, nvte_gemm_reduce_scatter and nvte_gemm_all_reduce had no parameter for it, and cuBLASMp had no attribute for it. FP8 forward GEMMs therefore ran with split accumulation on Hopper, while the cuBLASLt and Userbuffers paths use fast accumulation there. cuBLASMp 0.12.0 adds CUBLASMP_MATMUL_DESCRIPTOR_ATTRIBUTE_FAST_ACCUM. Add use_split_accumulator to the three functions, after accumulate as in nvte_cublas_gemm, and pass the value that the comm+GEMM overlap already receives. When A and B are FP8 and split accumulation is not requested, the GEMM sets the attribute, as cublaslt_gemm.cu does with cuBLASLt. With cuBLASMp older than 0.12.0, at build time or at run time (nvte_cublasmp_version()), the GEMMs keep split accumulation. The C++ comm GEMM test passes the setting to the overlapped GEMM and runs one FP8 case per collective with fast accumulation. Its cuBLASLt reference keeps split accumulation, so these cases check the accuracy as well. Signed-off-by: Almog Segal <asegal@nvidia.com>
[Medium impact] The PR appears safe to merge; no actionable issues were found. SummaryAdds
Reviews (1) · Last reviewed commit: "[Common] Use FP8 fast accumulation in th..." · Reviewed by Greptile |
timmoon10
left a comment
There was a problem hiding this comment.
Is there a particular FP8 CS workload on Hopper we are targetting with this? It's not relevant to more modern GPUs or quant recipes, so we're heading toward the direction of removing the option entirely (see #3752 (comment)).
There was a problem hiding this comment.
These changes break backward compatibility. If we want to expose a new option (which we might not), I'd suggest creating separate _v2 functions and creating an opaque config object. For example, see nvte_cublas_gemm_v2:
TransformerEngine/transformer_engine/common/include/transformer_engine/gemm.h
Lines 211 to 213 in 3b1fb58
This way we can add additional arguments without needing to create more _v3, _v4, _v5 variants.
Alp said on #3629 (review) that TE wants to be able to use the fast accumulator in the forward pass for FP8 so I added support for this in cuBLASMp 0.12.0 and this PR enables this on the TE side. |
The cuBLASMp comm+GEMM backend ignored use_split_accumulator: nvte_all_gather_gemm, nvte_gemm_reduce_scatter and nvte_gemm_all_reduce had no parameter for it, and cuBLASMp had no attribute for it. FP8 forward GEMMs therefore ran with split accumulation on Hopper, while the cuBLASLt and Userbuffers paths use fast accumulation there. cuBLASMp 0.12.0 adds CUBLASMP_MATMUL_DESCRIPTOR_ATTRIBUTE_FAST_ACCUM.
Add use_split_accumulator to the three functions, after accumulate as in nvte_cublas_gemm, and pass the value that the comm+GEMM overlap already receives. When A and B are FP8 and split accumulation is not requested, the GEMM sets the attribute, as cublaslt_gemm.cu does with cuBLASLt. With cuBLASMp older than 0.12.0, at build time or at run time (nvte_cublasmp_version()), the GEMMs keep split accumulation.
The C++ comm GEMM test passes the setting to the overlapped GEMM and runs one FP8 case per collective with fast accumulation. Its cuBLASLt reference keeps split accumulation, so these cases check the accuracy as well.
Description
Please include a brief summary of the changes, relevant motivation and context.
Fixes # (issue)
Type of change
Changes
Please list the changes introduced in this PR:
Checklist: