Skip to content

[WIP][Distributed] gin_overlap 1/6: wrap the NCCL device-API host entry points in dynload - #79966

Closed
AlAuAu wants to merge 3 commits into
PaddlePaddle:developfrom
AlAuAu:gin_split_1_dynload
Closed

AlAuAu wants to merge 3 commits into
PaddlePaddle:developfrom
AlAuAu:gin_split_1_dynload

Conversation

@AlAuAu

@AlAuAu AlAuAu commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

PR Category

Distributed Strategy

PR Types

New features

Description

Merged into #79967.

This is the first layer of the series; Files changed shows only this layer.

Part 1/6 of the gin_overlap primitive (#79964), split for review. Compiled with the gate on (CUDA 13.2, NCCL 2.31.2, CUDA_ARCH_BIN="90 100 103"); not run on a GPU yet.

The six PRs of this series: #79966 → #79967 → #79968 → #79969 → #79970 → #79971. Each branch contains the ones before it, so on GitHub every PR after the first shows the earlier layers too; review only the commits of this layer (the Commits tab), and merge in order.

Series: #79964 is the whole change; it is split into six PRs that each build on the previous one.
The six branches are identical to #79964 in total (empty git diff between the last one and #79964).

# Branch Adds
1 gin_split_1_dynload dynload wrappers (this PR)
2 gin_split_2_kernels CUDA kernels + CMake build gate
3 gin_split_3_engine_init engine setup: comms, windows, devcomm, side streams
4 gin_split_4_engine_submit per-stage submit functions, 2D-bucket gather
5 gin_split_5_release release layer (release.cc) and its engine wrappers
6 gin_split_6_python pybind module paddle.base.core.gin_overlap + Python layer

What this PR adds

Paddle links neither libnccl nor libcuda, so new code that calls them has to go through the
phi::dynload dlopen wrappers. The gin_overlap code in the following PRs needs three symbols
that are not wrapped yet:

  • ncclDevCommCreate / ncclGetLsaDevicePointer (paddle/phi/backends/dynload/nccl.{h,cc}): the two
    host entry points of NCCL's device (LSA/GIN) API. They are declared in nccl_device.h, which also
    carries the device-side inline implementation of the whole device API, so nccl.h repeats the two
    declarations instead of including it (the same way the internal entry points at the top of the file
    are declared), and skips them in translation units that include nccl_device.h themselves.
    Same 2.30.7 floor as the zero-SM group above them.
  • cuMemcpy2DAsync (paddle/phi/backends/dynload/cuda_driver.{h,cc}): wrapped under the name libcuda
    exports, cuMemcpy2DAsync_v2. The wrapper dlsym()s the name it is given but types itself from
    decltype(&::name), which cuda.h's macro resolves to the v2 declaration; passing the plain name
    would call the v1 entry point through a v2 signature. Call sites keep writing cuMemcpy2DAsync.

The host overload of ncclTeamWorld is not wrapped: it shares its name with the device-side
ncclTeamWorld(ncclDevComm const&), so decltype(&::ncclTeamWorld) cannot be formed, and no host
code needs it.

Depends on: develop (63c4ef64e6). Nothing else.

Callers after merge: none in this PR. The first caller is gin_overlap/gin_dynload.h (#79967),
used from #79968 on.

What CI can show: the cuMemcpy2DAsync_v2 wrapper is compiled on every CUDA build, so CI
shows it compiles. The two NCCL wrappers sit behind #if NCCL_VERSION_CODE >= 23007; they are
only compiled where the NCCL headers are 2.30.7 or newer. I have not checked which CI images
have such headers (setup.py pins nvidia-nccl-cu12 at 2.25.1 to 2.28.3), so CI may not compile
them at all. No CI job calls any of the three.

How correctness is established: the wrappers are only exercised by the gin_overlap engine
(#79968 onwards). The bitwise evidence so far is for earlier, out-of-tree states of that engine
(4 nodes / 32 cards, 10 steps, identical to Paddle's reduce_scatter baseline; see #79967), not for
this tree. The acceptance gate is the same 10-step AdamW and Muon run on a build of the whole
series. The whole series has been compiled (see #79967); nothing has been run on a GPU yet.

Changes after the first review

  • ncclDevCommCreate / ncclGetLsaDevicePointer joined NCCL_RAND_ROUTINE_EACH_AFTER_23007 and cuMemcpy2DAsync_v2 joined CUDA_ROUTINE_EACH; the two separate groups and their expansions in the .cc files are gone.
  • The comment on the nccl_device.h prototypes now says what holds: they must match nccl_device.h exactly, because this header is usually included first and both declarations then coexist; the guard only drops the duplicate in the other include order.

Not verified: none of these changes has run on a GPU. Every change is outside the numerical path (no kernel body, summation order, __fmul_rn / __fadd_rn order, release condition, credit target or coefficient placement changed); the 4-node bitwise acceptance run is still to come. Comments, docstrings, errors and logs of the series are now in English (one commit per layer; outside string literals the token stream / AST is identical, checked mechanically).

Changes after the second review

  • The comment of NCCL_RAND_ROUTINE_EACH_AFTER_23007 says that ncclDevCommCreate / ncclGetLsaDevicePointer are the host entry points of the device (LSA/GIN) API, also first in 2.30.7, so they share the gate of the zero-SM entry points. Comment only.

Not verified: the precision acceptance run (4 nodes, bitwise) is still in progress, and the changes of this round have not been through it. The behavior changes of this round are the argument check in release::setup (#79970); the cpplint fixes (#79967) change casts, integer type spellings and reference -> pointer parameters only.

是否引起精度变化

否

🤖 Generated with Claude Code

@Paddle-Review-Bot

Paddle-Review-Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Paddle-Bot Review Board (review完成)

序号 位置 优先级 规则来源 状态

可维护性建议(总分 90.4 / 100,只给建议,不影响评审结论)

序号 位置 维度 状态
1 设备 API 声明块的 guard 前提 契约一致性 ✅
2 设备 API 包装分组命名与门控 约定偏离 ✅
3 单成员 MEMCPY 宏分组 抽象适配度 ✅
4 零 SM 分组注释未覆盖设备 API 中层优雅度 ✅
各维度得分(仅评本 PR 改动,点击展开)
85 初始化耦合
85 测试策略
86.5 中层优雅度
4
88 API 表面一致性
88 依赖健康度
88 未完成迁移
88 底层优雅度
90 跨模块架构
90 错误处理一致性
90 高层优雅度
91 约定偏离
2
92 抽象适配度
3
92 契约一致性
1
92 设计一致性
92 类型安全
93 AI 生成债务
95 逻辑清晰度
95 命名质量
97 包组织结构
— 鉴权一致性

Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully.

Comment on lines +56 to +60
// Declarations of the two host entry points of NCCL's device (LSA/GIN) API,
// which live in nccl_device.h. nccl_device.h must not be included here. In a
// translation unit that includes nccl_device.h, its include guard turns this
// block off and that header's own declarations are used.
#if NCCL_VERSION_CODE >= 23007 && !defined(_NCCL_DEVICE_CORE_H_)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(契约一致性,不阻塞合入):注释写“包含 nccl_device.h 的翻译单元里,其 include guard 会关掉这一块”,但 !defined(_NCCL_DEVICE_CORE_H_) 只在 nccl_device.h 先于本头被包含时成立。本头经 paddle/fluid/platform/enforce.h 等被大量间接包含,常见顺序反而是本块先生效、之后再 include nccl_device.h,此时两份声明同时存在;能编译是因为 C++ 允许签名一致的重复声明,并不是靠 guard。另外 _NCCL_DEVICE_CORE_H_ 是 NCCL 内部的 include guard,并非公开契约。

建议把注释改成真实的前提:这里的原型必须与 nccl_device.h 中的声明逐字一致(2.30.7 在 nccl_device/core.h,2.32.x 移到 nccl_device/host.h);guard 只是在 nccl_device.h 已被包含时省掉重复声明。

Comment thread paddle/phi/backends/dynload/nccl.h Outdated
// the ncclDevComm that kernels take; ncclGetLsaDevicePointer returns a peer's
// window address in this rank's address space. Requires NCCL >= 2.30.7.
#if NCCL_VERSION_CODE >= 23007
#define NCCL_RAND_ROUTINE_EACH_DEVICE_API(__macro) \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(约定偏离,不阻塞合入):本文件已有的按版本门控分组都用版本号命名(_AFTER_2212 … _AFTER_23007),新增的 NCCL_RAND_ROUTINE_EACH_DEVICE_API 是唯一按功能命名的分组,并且在 L176 又开了一个与 L159 完全相同的 #if NCCL_VERSION_CODE >= 23007,nccl.cc 里也要多展开一行。搜索 _AFTER_23007 时会漏掉这两个符号,读者也容易误以为两组的版本要求不同。

L60-73 的原型声明在 NCCL_RAND_ROUTINE_EACH_AFTER_23007 展开之前,所以可以直接把 ncclDevCommCreate / ncclGetLsaDevicePointer 追加到该组末尾(保留这里的注释),然后删掉这个独立分组和它在 nccl.cc 的展开。

// libcuda also exports a v1 entry point with a different parameter layout, so
// the wrapper must dlsym() the _v2 name. Call sites keep writing
// cuMemcpy2DAsync.
#define CUDA_ROUTINE_EACH_MEMCPY(__macro) __macro(cuMemcpy2DAsync_v2)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(抽象适配度,不阻塞合入):CUDA_ROUTINE_EACH_MEMCPY 只有 cuMemcpy2DAsync_v2 一个成员,也没有版本门控;同文件的单成员分组 CUDA_ROUTINE_EACH_CUDA_GRAPH_12 单独存在是因为要放在 #if CUDA_VERSION >= 12040 里。这个新分组只多出 cuda_driver.h:99 和 cuda_driver.cc:27 两处展开,而 _MEMCPY 这个名字会让人以为这里收拢了一类 memcpy 接口。

建议把 __macro(cuMemcpy2DAsync_v2); 并入 CUDA_ROUTINE_EACH(cuDeviceGet 之后),并把 L64-68 关于 _v2 名字的说明放在该条目上方,同时删掉这个宏和它的两处展开。

Comment on lines +172 to +173
__macro(ncclDevCommCreate); \
__macro(ncclGetLsaDevicePointer);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(中层优雅度,不阻塞合入):ncclDevCommCreate / ncclGetLsaDevicePointer 并入后,L156-161 的组注释仍写“zero-SM (NCCL_CTA_POLICY_ZERO) 通信路径需要的入口”,并逐个列出各符号的引入版本,没有提到这两个设备 API(LSA/GIN)入口。读者会以为 2.30.7 门槛只来自 zero-SM collective,以后裁剪或调整这一组门控时容易误判。

建议在组注释里补一句:这两个设备 API host 入口自 2.30.7 才有(声明见上方 L56-78 的原型块),所以和 zero-SM 入口共用同一门控。

Paddle links neither libnccl nor libcuda, so new code that calls them has to go
through the phi::dynload dlopen wrappers. The gin_overlap gradient
communication added in the following commits needs three symbols that are not
wrapped yet:

* ncclDevCommCreate / ncclGetLsaDevicePointer, the two host entry points of
  NCCL's device (LSA/GIN) API. They are declared in nccl_device.h, which also
  carries the device-side inline implementations of the whole device API, so
  including it from nccl.h would land all of that in every translation unit
  that reaches for any NCCL symbol. The declarations it needs are repeated
  here instead, the way the internal entry points at the top of the file
  already are, and are skipped in translation units that include
  nccl_device.h themselves. Same 2.30.7 floor as the zero-SM group above
  them: below that the feature is refused, not degraded.
* cuMemcpy2DAsync, for the pitched copy that splits a fused all_gather back
  into per-bucket storages. It is wrapped under the name libcuda exports,
  cuMemcpy2DAsync_v2: the wrapper dlsym()s the name it is given but types
  itself from decltype(&::name), which cuda.h's macro resolves to the v2
  declaration, so passing the plain name would call the v1 entry point (whose
  parameter is the old CUDA_MEMCPY2D_v1 layout) through a v2 signature. Call
  sites keep writing cuMemcpy2DAsync and let cuda.h's macro rewrite it.

The host overload of ncclTeamWorld is deliberately not wrapped: it shares a
name with the device-side ncclTeamWorld(ncclDevComm const&), so
decltype(&::ncclTeamWorld) cannot be formed, and no host code needs it.

Co-Authored-By: Claude Code <noreply@anthropic.com>
ncclDevCommCreate and ncclGetLsaDevicePointer need the same NCCL 2.30.7
as the other symbols of NCCL_RAND_ROUTINE_EACH_AFTER_23007, so they join
that group instead of a separate NCCL_RAND_ROUTINE_EACH_DEVICE_API under
an identical #if. cuMemcpy2DAsync_v2 has no version gate and joins
CUDA_ROUTINE_EACH; CUDA_ROUTINE_EACH_MEMCPY and its two expansions go.

The comment on the nccl_device.h prototypes now states what actually
holds: the prototypes must match nccl_device.h exactly, because this
header is usually included first and both declarations then coexist;
the guard only drops the duplicate in the other include order.

Co-Authored-By: Claude Code <noreply@anthropic.com>
….30.7 gate

Co-Authored-By: Claude Code <noreply@anthropic.com>
@AlAuAu
AlAuAu force-pushed the gin_split_1_dynload branch from e2584fd to 0d1744b Compare November 21, 2026 11:21
@AlAuAu AlAuAu closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants