Skip to content

[Common] Use FP8 fast accumulation in the cuBLASMp comm+GEMM overlap - #3719

Open
almogsegal wants to merge 1 commit into
NVIDIA:mainfrom
almogsegal:cublasmp-fast-accum
Open

almogsegal wants to merge 1 commit into
NVIDIA:mainfrom
almogsegal:cublasmp-fast-accum

Conversation

@almogsegal

Copy link
Copy Markdown
Contributor

The cuBLASMp comm+GEMM backend ignored use_split_accumulator: nvte_all_gather_gemm, nvte_gemm_reduce_scatter and nvte_gemm_all_reduce had no parameter for it, and cuBLASMp had no attribute for it. FP8 forward GEMMs therefore ran with split accumulation on Hopper, while the cuBLASLt and Userbuffers paths use fast accumulation there. cuBLASMp 0.12.0 adds CUBLASMP_MATMUL_DESCRIPTOR_ATTRIBUTE_FAST_ACCUM.

Add use_split_accumulator to the three functions, after accumulate as in nvte_cublas_gemm, and pass the value that the comm+GEMM overlap already receives. When A and B are FP8 and split accumulation is not requested, the GEMM sets the attribute, as cublaslt_gemm.cu does with cuBLASLt. With cuBLASMp older than 0.12.0, at build time or at run time (nvte_cublasmp_version()), the GEMMs keep split accumulation.

The C++ comm GEMM test passes the setting to the overlapped GEMM and runs one FP8 case per collective with fast accumulation. Its cuBLASLt reference keeps split accumulation, so these cases check the accuracy as well.

Description

Please include a brief summary of the changes, relevant motivation and context.

Fixes # (issue)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Change A
  • Change B

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

The cuBLASMp comm+GEMM backend ignored use_split_accumulator: nvte_all_gather_gemm, nvte_gemm_reduce_scatter
and nvte_gemm_all_reduce had no parameter for it, and cuBLASMp had no attribute for it. FP8 forward GEMMs
therefore ran with split accumulation on Hopper, while the cuBLASLt and Userbuffers paths use fast accumulation
there. cuBLASMp 0.12.0 adds CUBLASMP_MATMUL_DESCRIPTOR_ATTRIBUTE_FAST_ACCUM.

Add use_split_accumulator to the three functions, after accumulate as in nvte_cublas_gemm, and pass the value
that the comm+GEMM overlap already receives. When A and B are FP8 and split accumulation is not requested, the
GEMM sets the attribute, as cublaslt_gemm.cu does with cuBLASLt. With cuBLASMp older than 0.12.0, at build time
or at run time (nvte_cublasmp_version()), the GEMMs keep split accumulation.

The C++ comm GEMM test passes the setting to the overlapped GEMM and runs one FP8 case per collective with fast
accumulation. Its cuBLASLt reference keeps split accumulation, so these cases check the accuracy as well.

Signed-off-by: Almog Segal <asegal@nvidia.com>
@almogsegal
almogsegal requested a review from ptrendx as a code owner October 21, 2026 06:43
@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Oct 21, 2026
@greptile-apps

greptile-apps Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were found.

Summary

Adds use_split_accumulator to all three cuBLASMp comm+GEMM functions and passes the existing overlap setting through to them.

  • FP8 overlap GEMMs can use fast accumulation on supported cuBLASMp versions.

Reviews (1) · Last reviewed commit: "[Common] Use FP8 fast accumulation in th..." · Reviewed by Greptile

@timmoon10 timmoon10 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a particular FP8 CS workload on Hopper we are targetting with this? It's not relevant to more modern GPUs or quant recipes, so we're heading toward the direction of removing the option entirely (see #3752 (comment)).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes break backward compatibility. If we want to expose a new option (which we might not), I'd suggest creating separate _v2 functions and creating an opaque config object. For example, see nvte_cublas_gemm_v2:

void nvte_cublas_gemm_v2(int transa, int transb, const float *alpha, const NVTETensor A,
const NVTETensor B, const float *beta, const NVTETensor C, NVTETensor D,
NVTETensor workspace, NVTEMatmulConfig config, cudaStream_t stream);

This way we can add additional arguments without needing to create more _v3, _v4, _v5 variants.

@almogsegal

Copy link
Copy Markdown
Contributor Author

Is there a particular FP8 CS workload on Hopper we are targetting with this? It's not relevant to more modern GPUs or quant recipes, so we're heading toward the direction of removing the option entirely (see #3752 (comment)).

Alp said on #3629 (review) that TE wants to be able to use the fast accumulator in the forward pass for FP8 so I added support for this in cuBLASMp 0.12.0 and this PR enables this on the TE side.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants