Repository navigation
Conversation
Paddle-Bot Review Board (review完成)
可维护性建议(总分 90.4 / 100,只给建议,不影响评审结论)
各维度得分(仅评本 PR 改动,点击展开)
Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully. |
| // Declarations of the two host entry points of NCCL's device (LSA/GIN) API, | ||
| // which live in nccl_device.h. nccl_device.h must not be included here. In a | ||
| // translation unit that includes nccl_device.h, its include guard turns this | ||
| // block off and that header's own declarations are used. | ||
| #if NCCL_VERSION_CODE >= 23007 && !defined(_NCCL_DEVICE_CORE_H_) |
There was a problem hiding this comment.
💡 可维护性建议(契约一致性,不阻塞合入):注释写“包含 nccl_device.h 的翻译单元里,其 include guard 会关掉这一块”,但 !defined(_NCCL_DEVICE_CORE_H_) 只在 nccl_device.h 先于本头被包含时成立。本头经 paddle/fluid/platform/enforce.h 等被大量间接包含,常见顺序反而是本块先生效、之后再 include nccl_device.h,此时两份声明同时存在;能编译是因为 C++ 允许签名一致的重复声明,并不是靠 guard。另外 _NCCL_DEVICE_CORE_H_ 是 NCCL 内部的 include guard,并非公开契约。
建议把注释改成真实的前提:这里的原型必须与 nccl_device.h 中的声明逐字一致(2.30.7 在 nccl_device/core.h,2.32.x 移到 nccl_device/host.h);guard 只是在 nccl_device.h 已被包含时省掉重复声明。
| // the ncclDevComm that kernels take; ncclGetLsaDevicePointer returns a peer's | ||
| // window address in this rank's address space. Requires NCCL >= 2.30.7. | ||
| #if NCCL_VERSION_CODE >= 23007 | ||
| #define NCCL_RAND_ROUTINE_EACH_DEVICE_API(__macro) \ |
There was a problem hiding this comment.
💡 可维护性建议(约定偏离,不阻塞合入):本文件已有的按版本门控分组都用版本号命名(_AFTER_2212 … _AFTER_23007),新增的 NCCL_RAND_ROUTINE_EACH_DEVICE_API 是唯一按功能命名的分组,并且在 L176 又开了一个与 L159 完全相同的 #if NCCL_VERSION_CODE >= 23007,nccl.cc 里也要多展开一行。搜索 _AFTER_23007 时会漏掉这两个符号,读者也容易误以为两组的版本要求不同。
L60-73 的原型声明在 NCCL_RAND_ROUTINE_EACH_AFTER_23007 展开之前,所以可以直接把 ncclDevCommCreate / ncclGetLsaDevicePointer 追加到该组末尾(保留这里的注释),然后删掉这个独立分组和它在 nccl.cc 的展开。
| // libcuda also exports a v1 entry point with a different parameter layout, so | ||
| // the wrapper must dlsym() the _v2 name. Call sites keep writing | ||
| // cuMemcpy2DAsync. | ||
| #define CUDA_ROUTINE_EACH_MEMCPY(__macro) __macro(cuMemcpy2DAsync_v2) |
There was a problem hiding this comment.
💡 可维护性建议(抽象适配度,不阻塞合入):CUDA_ROUTINE_EACH_MEMCPY 只有 cuMemcpy2DAsync_v2 一个成员,也没有版本门控;同文件的单成员分组 CUDA_ROUTINE_EACH_CUDA_GRAPH_12 单独存在是因为要放在 #if CUDA_VERSION >= 12040 里。这个新分组只多出 cuda_driver.h:99 和 cuda_driver.cc:27 两处展开,而 _MEMCPY 这个名字会让人以为这里收拢了一类 memcpy 接口。
建议把 __macro(cuMemcpy2DAsync_v2); 并入 CUDA_ROUTINE_EACH(cuDeviceGet 之后),并把 L64-68 关于 _v2 名字的说明放在该条目上方,同时删掉这个宏和它的两处展开。
928787b to
e2584fd
Compare
| __macro(ncclDevCommCreate); \ | ||
| __macro(ncclGetLsaDevicePointer); |
There was a problem hiding this comment.
💡 可维护性建议(中层优雅度,不阻塞合入):ncclDevCommCreate / ncclGetLsaDevicePointer 并入后,L156-161 的组注释仍写“zero-SM (NCCL_CTA_POLICY_ZERO) 通信路径需要的入口”,并逐个列出各符号的引入版本,没有提到这两个设备 API(LSA/GIN)入口。读者会以为 2.30.7 门槛只来自 zero-SM collective,以后裁剪或调整这一组门控时容易误判。
建议在组注释里补一句:这两个设备 API host 入口自 2.30.7 才有(声明见上方 L56-78 的原型块),所以和 zero-SM 入口共用同一门控。
Paddle links neither libnccl nor libcuda, so new code that calls them has to go through the phi::dynload dlopen wrappers. The gin_overlap gradient communication added in the following commits needs three symbols that are not wrapped yet: * ncclDevCommCreate / ncclGetLsaDevicePointer, the two host entry points of NCCL's device (LSA/GIN) API. They are declared in nccl_device.h, which also carries the device-side inline implementations of the whole device API, so including it from nccl.h would land all of that in every translation unit that reaches for any NCCL symbol. The declarations it needs are repeated here instead, the way the internal entry points at the top of the file already are, and are skipped in translation units that include nccl_device.h themselves. Same 2.30.7 floor as the zero-SM group above them: below that the feature is refused, not degraded. * cuMemcpy2DAsync, for the pitched copy that splits a fused all_gather back into per-bucket storages. It is wrapped under the name libcuda exports, cuMemcpy2DAsync_v2: the wrapper dlsym()s the name it is given but types itself from decltype(&::name), which cuda.h's macro resolves to the v2 declaration, so passing the plain name would call the v1 entry point (whose parameter is the old CUDA_MEMCPY2D_v1 layout) through a v2 signature. Call sites keep writing cuMemcpy2DAsync and let cuda.h's macro rewrite it. The host overload of ncclTeamWorld is deliberately not wrapped: it shares a name with the device-side ncclTeamWorld(ncclDevComm const&), so decltype(&::ncclTeamWorld) cannot be formed, and no host code needs it. Co-Authored-By: Claude Code <noreply@anthropic.com>
ncclDevCommCreate and ncclGetLsaDevicePointer need the same NCCL 2.30.7 as the other symbols of NCCL_RAND_ROUTINE_EACH_AFTER_23007, so they join that group instead of a separate NCCL_RAND_ROUTINE_EACH_DEVICE_API under an identical #if. cuMemcpy2DAsync_v2 has no version gate and joins CUDA_ROUTINE_EACH; CUDA_ROUTINE_EACH_MEMCPY and its two expansions go. The comment on the nccl_device.h prototypes now states what actually holds: the prototypes must match nccl_device.h exactly, because this header is usually included first and both declarations then coexist; the guard only drops the duplicate in the other include order. Co-Authored-By: Claude Code <noreply@anthropic.com>
….30.7 gate Co-Authored-By: Claude Code <noreply@anthropic.com>
e2584fd to
0d1744b
Compare
PR Category
Distributed Strategy
PR Types
New features
Description
Merged into #79967.
This is the first layer of the series; Files changed shows only this layer.
Part 1/6 of the gin_overlap primitive (#79964), split for review. Compiled with the gate on (CUDA 13.2, NCCL 2.31.2,
CUDA_ARCH_BIN="90 100 103"); not run on a GPU yet.The six PRs of this series: #79966 → #79967 → #79968 → #79969 → #79970 → #79971. Each branch contains the ones before it, so on GitHub every PR after the first shows the earlier layers too; review only the commits of this layer (the Commits tab), and merge in order.
Series: #79964 is the whole change; it is split into six PRs that each build on the previous one.
The six branches are identical to #79964 in total (empty
git diffbetween the last one and #79964).gin_split_1_dynloadgin_split_2_kernelsgin_split_3_engine_initgin_split_4_engine_submitgin_split_5_releaserelease.cc) and its engine wrappersgin_split_6_pythonpaddle.base.core.gin_overlap+ Python layerWhat this PR adds
Paddle links neither libnccl nor libcuda, so new code that calls them has to go through the
phi::dynloaddlopen wrappers. The gin_overlap code in the following PRs needs three symbolsthat are not wrapped yet:
ncclDevCommCreate/ncclGetLsaDevicePointer(paddle/phi/backends/dynload/nccl.{h,cc}): the twohost entry points of NCCL's device (LSA/GIN) API. They are declared in
nccl_device.h, which alsocarries the device-side inline implementation of the whole device API, so
nccl.hrepeats the twodeclarations instead of including it (the same way the internal entry points at the top of the file
are declared), and skips them in translation units that include
nccl_device.hthemselves.Same 2.30.7 floor as the zero-SM group above them.
cuMemcpy2DAsync(paddle/phi/backends/dynload/cuda_driver.{h,cc}): wrapped under the name libcudaexports,
cuMemcpy2DAsync_v2. The wrapperdlsym()s the name it is given but types itself fromdecltype(&::name), whichcuda.h's macro resolves to the v2 declaration; passing the plain namewould call the v1 entry point through a v2 signature. Call sites keep writing
cuMemcpy2DAsync.The host overload of
ncclTeamWorldis not wrapped: it shares its name with the device-sidencclTeamWorld(ncclDevComm const&), sodecltype(&::ncclTeamWorld)cannot be formed, and no hostcode needs it.
Depends on: develop (
63c4ef64e6). Nothing else.Callers after merge: none in this PR. The first caller is
gin_overlap/gin_dynload.h(#79967),used from #79968 on.
What CI can show: the
cuMemcpy2DAsync_v2wrapper is compiled on every CUDA build, so CIshows it compiles. The two NCCL wrappers sit behind
#if NCCL_VERSION_CODE >= 23007; they areonly compiled where the NCCL headers are 2.30.7 or newer. I have not checked which CI images
have such headers (
setup.pypinsnvidia-nccl-cu12at 2.25.1 to 2.28.3), so CI may not compilethem at all. No CI job calls any of the three.
How correctness is established: the wrappers are only exercised by the gin_overlap engine
(#79968 onwards). The bitwise evidence so far is for earlier, out-of-tree states of that engine
(4 nodes / 32 cards, 10 steps, identical to Paddle's
reduce_scatterbaseline; see #79967), not forthis tree. The acceptance gate is the same 10-step AdamW and Muon run on a build of the whole
series. The whole series has been compiled (see #79967); nothing has been run on a GPU yet.
Changes after the first review
ncclDevCommCreate/ncclGetLsaDevicePointerjoinedNCCL_RAND_ROUTINE_EACH_AFTER_23007andcuMemcpy2DAsync_v2joinedCUDA_ROUTINE_EACH; the two separate groups and their expansions in the.ccfiles are gone.nccl_device.hprototypes now says what holds: they must matchnccl_device.hexactly, because this header is usually included first and both declarations then coexist; the guard only drops the duplicate in the other include order.Not verified: none of these changes has run on a GPU. Every change is outside the numerical path (no kernel body, summation order,
__fmul_rn/__fadd_rnorder, release condition, credit target or coefficient placement changed); the 4-node bitwise acceptance run is still to come. Comments, docstrings, errors and logs of the series are now in English (one commit per layer; outside string literals the token stream / AST is identical, checked mechanically).Changes after the second review
NCCL_RAND_ROUTINE_EACH_AFTER_23007says thatncclDevCommCreate/ncclGetLsaDevicePointerare the host entry points of the device (LSA/GIN) API, also first in 2.30.7, so they share the gate of the zero-SM entry points. Comment only.Not verified: the precision acceptance run (4 nodes, bitwise) is still in progress, and the changes of this round have not been through it. The behavior changes of this round are the argument check in
release::setup(#79970); the cpplint fixes (#79967) change casts, integer type spellings and reference -> pointer parameters only.是否引起精度变化
否
🤖 Generated with Claude Code