Repository navigation
Conversation
Paddle links neither libnccl nor libcuda, so new code that calls them has to go through the phi::dynload dlopen wrappers. The gin_overlap gradient communication added in the following commits needs three symbols that are not wrapped yet: * ncclDevCommCreate / ncclGetLsaDevicePointer, the two host entry points of NCCL's device (LSA/GIN) API. They are declared in nccl_device.h, which also carries the device-side inline implementations of the whole device API, so including it from nccl.h would land all of that in every translation unit that reaches for any NCCL symbol. The declarations it needs are repeated here instead, the way the internal entry points at the top of the file already are, and are skipped in translation units that include nccl_device.h themselves. Same 2.30.7 floor as the zero-SM group above them: below that the feature is refused, not degraded. * cuMemcpy2DAsync, for the pitched copy that splits a fused all_gather back into per-bucket storages. It is wrapped under the name libcuda exports, cuMemcpy2DAsync_v2: the wrapper dlsym()s the name it is given but types itself from decltype(&::name), which cuda.h's macro resolves to the v2 declaration, so passing the plain name would call the v1 entry point (whose parameter is the old CUDA_MEMCPY2D_v1 layout) through a v2 signature. Call sites keep writing cuMemcpy2DAsync and let cuda.h's macro rewrite it. The host overload of ncclTeamWorld is deliberately not wrapped: it shares a name with the device-side ncclTeamWorld(ncclDevComm const&), so decltype(&::ncclTeamWorld) cannot be formed, and no host code needs it. Co-Authored-By: Claude Code <noreply@anthropic.com>
…nto Paddle Each sharding-stage1 gradient bucket normally costs one NCCL reduce_scatter. gin_overlap replaces that with NCCL GIN one-sided puts done in several steps, so every step can be slotted into the gaps in the backward pass where DeepEP is using neither the network nor the SMs. The code has been living in a PaddleFleet extension (paddlefleet_ops/gin_overlap/csrc); the primitives belong next to Paddle's other one-sided communication, while the policy of *when* to advance which stage stays in the training framework. This is a move plus wiring, not a rewrite: the sum order, the kernels, the release conditions, the block counts, the scale factors and the `stages` semantics of advance() are untouched, and the comments and self-checks came along with them. Per file the diff against the extension is only include paths, namespaces, include guards (#pragma once) and file names in comments. Two exceptions: * paddle_comm.cu and paddle_tensor.cu become .cc. They were .cu only because an out-of-tree build has to hand any translation unit that includes Paddle headers to nvcc (thrust); in-tree they are ordinary sources. The pybind registration that was stranded in paddle_tensor.cu moves to the pybind layer in the next commit, so paddle_tensor.h can declare the real signatures (paddle::Tensor) instead of a "go register over there" hook. The zero-copy contract is unchanged: paddle::from_blob with deleter = nullptr, so phi::Allocation::deleter_ stays null and the window memory is never freed by Paddle, plus the same post-wrap self-check. * reduce.cu's "must stay bit-for-bit" list loses its block/thread item. It contradicts the code: the scheduler changes the reduce kernels' block count between windows within one run (REDUCE_BLOCKS / FINALIZE_BLOCKS), and the bit-exactness check against reduce_scatter has been passing all along. The block count only decides which indices a block walks, not the order the sources are summed in. The other items stay. Two casts of CUDA built-ins are respelled as static_cast so that the no-int64-type-cuda-index-without-cast ast-grep rule sees them (the C-style cast was already on the first operand, so this changes nothing); cpplint is switched off for the directory, as it is for deep_ep, rather than rewriting every cast and `long long` in moved code. Build gating (CMake is written but not run in this change): * arch: only built when NVCC_ARCH_BIN contains 9.0 or newer, computed the way paddle/phi/kernels/CMakeLists.txt computes has_arch_ge80. Inside the directory the <= 8.x gencodes are stripped off CMAKE_CUDA_FLAGS, as deep_ep/CMakeLists.txt does, so no device code is built for cards this is not supported on. * NCCL >= 2.30.7 with nccl_device.h present, and CUDA >= 12.8 for cudaMemcpyBatchAsync. The result is cached as WITH_GIN_OVERLAP so the pybind layer does not restate the condition. Unsupported configurations keep raising rather than degrading: gin_dynload.h #errors below the NCCL floor, and the Python layer reports a missing module instead of a half-built one. Co-Authored-By: Claude Code <noreply@anthropic.com>
The extension's module (paddlefleet_ops.gin_overlap_cpp, built from bindings.cpp) becomes a submodule of libpaddle, registered the way deep_ep_api.cc and cudart_py.cc register theirs: a BindGinOverlapApi(&m) call from pybind.cc, the whole file empty unless PADDLE_WITH_GIN_OVERLAP is defined, and that definition driven by the WITH_GIN_OVERLAP the collective CMakeLists computed. A submodule rather than attributes on core directly: besides GradCommEngine there are 24 release-layer constants, and paddle.base.core is not the place for names like ADVANCE_PUT. Method names, py::arg names and return conventions are unchanged -- the Python side calls them by name, and the release-layer constants still have their single definition in release.cu (release::constants), from where this file publishes them both as submodule attributes and through GradCommEngine.release_constants(), which the caller checks name-by-name at startup. The two methods that take or return paddle::Tensor (send_window_tensor, copy_2d) are registered here too, instead of from a separate .cu as the extension had to: the caster in paddle/utils/pybind.h is usable from this layer, which is what deep_ep_api.cc relies on as well. cpplint on this file: the include order is fixed and the two casts of the group ids are respelled static_cast, but the `long long` parameters only get a NOLINT. They have to match the C++ signatures word for word -- on LP64 int64_t is long, so std::vector<long long> and std::vector<int64_t> are different types and the out-parameter calls would not compile. Co-Authored-By: Claude Code <noreply@anthropic.com>
The thin Python half of the primitive: ~900 lines moved almost verbatim out of PaddleFleet's engine.py into GinOverlapPrimitive. It computes the layouts (send window per bucket, the three receive-pool rings, the host-window counter blocks), builds the two communication groups, hands the release layer its per-bucket parameter table, builds the 2D-bucket gather table, and prints the init log. Kept in Python on purpose: it is startup-time arithmetic plus collective cross-rank agreement (all_reduce of bounds, crc of the tables), which reads far better here than in C++, and it is not on any per-step path. Placed next to deep_ep, under paddle/distributed/communication, for the same reason: a Python shell over a CUDA implementation reached through paddle.base.core. The split line is "layout and how to hand it to C++" here versus "when to advance, and talking to the sharding optimizer" in the caller, which subclasses this (PaddleFleet's GinOverlapEngine). Three edits make the class free of the caller: the receive-pool sizes arrive as constructor arguments instead of being read off PaddleFleet's config object, the scheduler / DeepEP-hook / barrier tail of __init__ moves to the subclass (which picks the timing list up from self._init_times), and _log_init takes an `extra` string so the caller's one config line still lands in the same log line it used to. The 2D-gather setup splits in two: the table build, crc check and handoff to C++ stay here as gather_table_setup, the subclass wraps it to attach its scheduler. Every cross-boundary name was reconciled statically against the pybind registrations (40 exported methods, 39 called, names and argument counts all matching) -- nothing in here has been compiled or run. Co-Authored-By: Claude Code <noreply@anthropic.com>
Paddle-Bot Review Board (review完成)可维护性建议(总分 87.3 / 100,只给建议,不影响评审结论)
各维度得分(仅评本 PR 改动,点击展开)
Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully. |
| target_compile_options( | ||
| gin_overlap PRIVATE -Wno-unused-variable -Wno-unused-but-set-variable | ||
| -Wno-sign-compare -Wno-reorder) |
There was a problem hiding this comment.
gin_overlap 目标的 SRCS 里有 kernels/*.cu,cc_library 会把它们按 CUDA 语言交给 nvcc;target_compile_options 没区分语言,这四个 -Wno-* 也会原样传给 nvcc。nvcc 不认识这些选项,直接报 nvcc fatal : Unknown option '-Wno-unused-variable'(-Wno-reorder 同样会报错)。结果是只要门控打开(也就是这个功能唯一有意义的构建配置),编译就会失败。deep_ep 能这样写,是因为它的 target_compile_options 只加在纯 .cc/.cpp 的 deep_ep 目标上,.cu 放在单独的 deepep_kernels 目标里。
建议用 $<$<COMPILE_LANGUAGE:CXX>:-Wno-unused-variable ...> 包起来;如果 .cu 也需要降级,就用 -Xcompiler= 单独传。另一种做法是照 deep_ep 把 kernels 拆成单独的目标。
There was a problem hiding this comment.
更正:这条判断不成立。CMake 对 nvcc ≥ 10.2 会在每条 CUDA 编译命令前加 -forward-unknown-to-host-compiler(Modules/Compiler/NVIDIA-CUDA.cmake),而门控要求 CUDA ≥ 13.0,所以 nvcc 不认识的 -Wno-* 会转交给 host 编译器,不会导致 .cu 编译失败。PR 描述里给出的 compile_commands.json 记录和构建结果也与此一致。这一项撤回,不需要修改。
There was a problem hiding this comment.
858be8c 把这几个 -Wno-* 限定为 $<COMPILE_LANGUAGE:CXX>,不再传给 nvcc;.cu 本身已由 cmake/cuda.cmake 加了 -w,告警行为不变。此项已解决。
| CUDA_CHECK(cudaMemcpyBatchAsync(dst_pointers.data(), | ||
| src_pointers.data(), | ||
| sizes.data(), | ||
| dst_pointers.size(), | ||
| ©_attrs, | ||
| &fail_index, | ||
| 1, | ||
| g_stream_intra_node_copy)); |
There was a problem hiding this comment.
这里(以及第 942 行)按 CUDA 13 的 8 参数签名调用 cudaMemcpyBatchAsync。但 CUDA 12.8/12.9 的签名有 9 个参数:(void **dsts, void **srcs, size_t *sizes, size_t count, cudaMemcpyAttributes *attrs, size_t *attrsIdxs, size_t numAttrs, size_t *failIdx, cudaStream_t stream),多一个 failIdx,srcs 也不是 const void**。现在的写法让 &fail_index 落在 attrsIdxs 的位置,参数个数和 const 都对不上,12.x 下编译不过。而 collective/CMakeLists.txt:48 的门控放行的是 CUDA_VERSION >= 12.8。
建议按 CUDART_VERSION 分两种调用(12.x 时补上 attrsIdxs,并用 const_cast 处理 srcs),或者把门控提到 13.0,同时改掉第 30-31 行的注释。
There was a problem hiding this comment.
已在 0e42587 修复:门控改为 CUDA_VERSION VERSION_GREATER_EQUAL 13.0,gin_dynload.h 增加了 CUDART_VERSION < 13000 的编译期检查。两处 cudaMemcpyBatchAsync 调用都符合 CUDA 13 的 8 参数签名。
| 不是公开 API:__all__ 故意是空的,用的人按名字从 .engine 里取。 | ||
| """ | ||
|
|
||
| from .engine import ( # noqa: F401 |
There was a problem hiding this comment.
新包 paddle.distributed.communication.gin_overlap 没有加进 setup.py(第 2641 行起)和 python/setup.py.in 的 packages 显式列表。这两处都不用 find_packages,不在列表里的目录不会打进 wheel。结果是 pip 安装后 paddle.base.core.gin_overlap 存在,Python 层 engine.py 却不在包里,PaddleFleet 无法 import。
建议照 deep_ep 的做法(setup.py:2846/2851、setup.py.in:1124/1130)在两个文件里都加上 packages.extend(['paddle.distributed.communication.gin_overlap'])。这个包是纯 Python,没编进 C++ 时 device_engine() 会给出原因,所以也可以无条件加入。
There was a problem hiding this comment.
已在 11b8fde 修复:setup.py 和 python/setup.py.in 都按 WITH_GIN_OVERLAP 条件加入了这个包,env_dict.py.in 也补上了对应变量。WITH_GIN_OVERLAP 是 paddle/ 目录下设置的 CACHE 变量,python/ 目录在其后 configure,取值正确。
| self.moe_bucket_numels = list(moe_numels) | ||
| self.expert_group_size = self.expert_group_rank = 0 | ||
| if self.moe_bucket_numels: | ||
| assert moe_group is not None, "有专家桶但没有专家通信组" |
There was a problem hiding this comment.
There was a problem hiding this comment.
已在 4310e17 修复:engine.py 里的输入校验都改成了 raise RuntimeError,不再依赖 assert。
| using gin_overlap::GradCommEngine; | ||
| #endif | ||
|
|
||
| void BindGinOverlapApi(py::module* m) { |
There was a problem hiding this comment.
这个 PR 新增了一整套分布式梯度通信原语(CUDA kernel、pybind、Python 布局计算),test/ 下却没有任何 gin_overlap 用例。
建议至少补两类测试:
_recv_pool_ring/_recv_pool_sizes/_send_layout/_topology这类纯整数布局计算可以写 CPU 单测。偏移算错时所有 rank 会得到同样的错误结果,不会当场崩溃,只会让数据写错位置。- 照
test/collective/test_collective_deep_ep_alltoall_intranode.py的方式加一个多进程用例,按 sm90 / NCCL 版本 skip,和 NCCL reduce_scatter/all_gather 做逐位比对,并登记到testslist.csv。
PR 描述里提到的 4 节点逐位验证也建议沉淀成可复跑的用例。
There was a problem hiding this comment.
4310e17 新增的 test_gin_overlap_layout.py 覆盖了 _send_layout、_recv_pool_ring 和 _recv_pool_sizes,CPU 布局测试这部分已满足。多进程用例(与 NCCL reduce_scatter / all_gather 逐位比对,按 sm90 / NCCL 版本 skip)仍然缺少,这条先保留。
There was a problem hiding this comment.
PR 描述说明多进程逐位比对测试会在 Paddle 侧合入后补充,多卡数值验证目前在 PaddleFleet 侧。这一项先保持打开:仓库规则要求分布式行为在 Paddle 内有代表性的多进程覆盖,外部仓库的测试不在本仓库 CI 中运行,无法防止后续改动造成回归。建议合入前至少在 test/collective/ 补一个与 reduce_scatter / all_gather 对比的多卡用例。
| size_t want_bytes, | ||
| uintptr_t* cpu_ptr, | ||
| size_t* nbytes); | ||
| int join_bucket_streams(int is_dense, int bucket); |
There was a problem hiding this comment.
💡 可维护性建议(命名质量,不阻塞合入):join_bucket_streams / gather_wait_bucket(第 467 行)的 is_dense 约定 1=dense、0=专家,而第 443 行的 bucket_ready 和放行层的 kind 约定 0=dense、1=专家(release.cu 里写的是 kind ? GROUP_MOE : GROUP_DENSE)。同一维度的取值方向相反,调用方把 kind 直接传给 is_dense 不会报错,只会把桶归错组。建议这两个方法统一改成收 int group(取 GROUP_DENSE / GROUP_MOE),pybind 和 Python 侧跟着改。
There was a problem hiding this comment.
已在 e297106 修改:join_bucket_streams / gather_wait_bucket 改为收 kind(0 = dense,1 = 专家),和 bucket_ready、放行层的约定一致,并拒绝 0 / 1 以外的取值。
| // 设备版同名重载,DECLARE_DYNAMIC_LOAD_NCCL_WRAP 里的 decltype(&::__name) | ||
| // 对重载名无法定型;host 侧也没有一处用它(put.cu / signal.cu 用的都是收 | ||
| // ncclDevComm 的设备版)。 | ||
| // * cudaMemcpyBatchAsync 等 runtime API 不用包:libcudart 是静态链进来的。 |
There was a problem hiding this comment.
💡 可维护性建议(依赖健康度,不阻塞合入):这条理由和实际构建不符。Linux 上顶层 CMakeLists.txt:113-117 设了 CUDA_USE_STATIC_CUDA_RUNTIME OFF 和 --cudart shared,libcudart 是动态链接的。因此 gin_engine.cc 里直接调用的 cudaMemcpyBatchAsync(CUDA 12.8 才有)会成为 libpaddle.so 对 cudart 版本的硬依赖。建议改成 phi::dynload 包装(或用 cudaGetDriverEntryPoint 取符号),缺失时给出明确报错;至少把注释改成真实的约束。
There was a problem hiding this comment.
已在 30cfa3a 修复:注释改为说明 runtime API 直接调用、Linux 上 libcudart 动态链接,并写明需要 CUDA ≥ 13.0,与实际构建一致。
|
|
||
| // 在 Paddle 当前计算流的当前位置记一个 mark,返回槽号(<0 = 出错) | ||
| int GradCommEngine::record_window_mark() { | ||
| if (g_window_marks.empty()) return -1; |
There was a problem hiding this comment.
💡 可维护性建议(错误处理一致性,不阻塞合入):gin_engine.h 和 gin_overlap_api.cc 都约定方法返回错误码(0 成功,<10000 为 cudaError_t),但 record_window_mark 成功时返回槽号、失败时返回 -1,tail_mark 也是透传。-1 交给 error_string 翻译不出原因,paddle_compute_stream 失败和 cudaEventRecord 失败也被合并成同一个 -1。建议槽号改走出参,返回值只放真实的 cudaError_t。
There was a problem hiding this comment.
ba847b3 补充了返回值说明,但槽号(≥ 0)和三种失败原因仍共用一个 -1,调用方无法区分失败原因,和其他方法“返回错误码”的约定也不一致。建议仍改为返回错误码、槽号通过出参返回。
There was a problem hiding this comment.
已在 e42c0ac 修复:record_window_mark / tail_mark 改为返回错误码、槽号通过出参返回,pybind 层失败时抛出带错误信息的 RuntimeError,并补了对应用例。
| paddle/utils/variant\.h| | ||
| paddle/utils/array_ref\.h| | ||
| paddle/fluid/distributed/collective/deep_ep/.*| | ||
| paddle/fluid/distributed/collective/gin_overlap/.*| |
There was a problem hiding this comment.
💡 可维护性建议(约定偏离,不阻塞合入):这个列表的注释是 “Exclude third-party libraries”,现有条目 deep_ep、deep_gemm 等都是从外部引入的代码。gin_overlap 是这个 PR 新写的一方代码,加进来后整个目录以后都不再经过 cpplint。建议去掉这一行,修掉 cpplint 报出的问题;确实需要豁免的个别规则,在对应行用 // NOLINT(<category>) 标注。
There was a problem hiding this comment.
已在 fb7228c 修复:.pre-commit-config.yaml 不再把 gin_overlap/ 排除在 cpplint 之外,目录内的告警也已按 cpplint 修正。
| f"gin_overlap: 取 Paddle 已有的 ncclComm 失败:{err or device_engine().error_string(rc)}" | ||
| ) | ||
|
|
||
| def _setup_group(self, gid, what, send_bytes, recv_bytes, signal_bytes, host_bytes, size, rank, rpn, ml, nn, mn, |
There was a problem hiding this comment.
💡 可维护性建议(抽象适配度,不阻塞合入):_setup_group 收 13 个位置参数,其中 size/rank/rpn/ml/nn/mn 六个是“一个组的拓扑”。两个调用点(第 235-245 行)只是在 self.group_* 和 self.moe_* 之间换着传,要靠 # fmt: skip 才能排下;_check_devcomm 又把同一组值再转一遍。参数顺序传错不会报错,只会用错组的拓扑。建议在 __init__ 里给 dense 和专家组各建一个小的 NamedTuple(gid、name、size、rank、ranks_per_node、rank_in_node、num_nodes、node_index、group),两个函数改为直接收它。
Only comments and docstrings change; no code, no string literal. - One full description per language: gin_engine.h file header (C++) and the engine.py module docstring (Python). Other files point there. - Comments now say what a file / function / layout is and what must hold, not where it came from: drop "originally X, now Y", DeepEP / PaddleFleet references, measurement anecdotes, cross-repo paths and file:line references. - Correct stale statements: the host window is read by the C++ release layer (release.cu), not by Python; the preload reason no longer cites memop waits; REDUCE_BLOCKS / FINALIZE_BLOCKS and "proxy" are gone. - One term per concept: "分片求和" (inter_node_reduce / moe_reduce, together shard_reduce), "尾部线程", "发 credit", "提交到 GPU". - Re-flow the blocks clang-format had folded (host window layout table etc.) into one item per line. Verified with comment_clean_check.py 6aed148..HEAD --base 63c4ef6: all 21 files OK (C++/CUDA token sequence identical after stripping comments, CMake tokens identical, Python AST identical after removing docstrings), and every changed line in shared files lies in lines added after 63c4ef6. pre-commit run --files on the changed files: all passed. Not compiled. Co-Authored-By: Claude Code <noreply@anthropic.com>
Paddle-Review-Bot
left a comment
There was a problem hiding this comment.
本次提交只改了注释和 docstring,代码未变。之前的 P0/P1 问题仍未修复,见各行内评论:gin_overlap 目标的 -Wno-* 选项会传给 nvcc;cudaMemcpyBatchAsync 在 CUDA 12.8/12.9 下签名不匹配;新 Python 包未加入打包列表。
… the wheel setup.py and python/setup.py.in list the packages that go into the wheel explicitly, and the gin_overlap Python layer was not among them, so a build with WITH_GIN_OVERLAP on shipped core.gin_overlap without the module that drives it. Add it under the same condition the C++ side is built under, passed through env_dict like COMPILED_CUDA_ARCHS. Co-Authored-By: Claude Code <noreply@anthropic.com>
Translate every comment, docstring, error message and log line of the gin_overlap sources (C++, CUDA, CMake, pybind and the Python layer) into English, and tighten the wording on the way. The terms are defined once, in the gin_engine.h file header (C++) and in the engine.py module docstring (Python); every other file uses them as defined there. Strings only. comment_clean_check.py --ignore-strings passes on every changed file: C++ / CUDA / CMake token streams are identical except for string literals (printf conversions inside merged literals keep their order), and the Python ASTs are identical except for str constants and the literal parts of f-strings. Log lines all carry the [gin_overlap] prefix; copy_2d errors now start with "copy_2d:". Co-Authored-By: Claude Code <noreply@anthropic.com>
ncclDevCommCreate and ncclGetLsaDevicePointer need the same NCCL 2.30.7 as the other symbols of NCCL_RAND_ROUTINE_EACH_AFTER_23007, so they join that group instead of a separate NCCL_RAND_ROUTINE_EACH_DEVICE_API under an identical #if. cuMemcpy2DAsync_v2 has no version gate and joins CUDA_ROUTINE_EACH; CUDA_ROUTINE_EACH_MEMCPY and its two expansions go. The comment on the nccl_device.h prototypes now states what actually holds: the prototypes must match nccl_device.h exactly, because this header is usually included first and both declarations then coexist; the guard only drops the duplicate in the other include order. Co-Authored-By: Claude Code <noreply@anthropic.com>
…the gate Both cudaMemcpyBatchAsync calls use the CUDA 13 signature (8 arguments, const void** sources). CUDA 12.8 / 12.9 declare a 9-argument version, so a build with the gate open on those toolkits would not compile. Raise the gate to CUDA >= 13.0, the only toolkit the code has been built and run with, instead of adding an untested 12.x branch; gin_dynload.h gets the matching #error backstop. The batch attribute index is named attrs_idx, which is what that argument is. The arch condition reuses ARCH_BIN_CONTAINS_GE_90, computed at the top of the same file for DeepEP, instead of a second loop over NVCC_ARCH_BIN. _BUILD_HINT in the Python layer now lists exactly the gate's conditions, and the CMake comment says to change both together. Co-Authored-By: Claude Code <noreply@anthropic.com>
…ne API
Argument checks (values come from Python):
- scale_range, gather_put_to_owner, send_window_tensor and copy_2d check
their bounds without unsigned wrap-around, and reject numel < 0.
copy_2d also rejects non-contiguous tensors (a stride-0 or transposed
view passes a numel-based bound but the copy reads contiguous bytes).
- release::setup checks every offset of the per-bucket tables and the
host window areas: non-negative, aligned (counters 8 B, data 4 B) and
inside its window. setup_gather_table checks both arrival counter
offsets of each entry the same way. CommGroup records recv_bytes for
this, and alloc_symmetric_window returns the size it actually
registered instead of the caller repeating the 4096 B minimum.
- A wrong state returns cudaErrorNotReady and a bad argument
cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
- bucket_ready rejects a bucket reported twice in a step in the RUN state
too, from a bitmap only the main thread touches.
Init failures:
- configure destroys the streams and events it created, frees its
temporary buffer and resets the globals it set when a later step fails,
so it can be retried. set_comms clears the groups it set on failure.
release::setup unpublishes its control block if the tail thread cannot
be started. The steps that cannot be rolled back (window registration,
devcomm creation, the host window) say so: restart the process.
- The gin_engine.h constraints state that the engine lives as long as
the process; shutdown_process_group / restart_process_group are not
supported.
API:
- GIN_CHECK replaces the dozen { rc = launch_x(...); if (rc) return rc; }
blocks. launch_* / preload_* return cudaError_t.
- launch_put_to_expert_peers takes (send_offset, recv_offset) like
launch_put_to_node_peers.
- join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert),
as bucket_ready and the gather table do, instead of is_dense.
- The six release positions are named variables, one per stage.
- The methods that do not return a code (record_window_mark, tail_mark,
bucket_ready, tail_all_posted, tail_wait, tail_abort, release_activity,
the version getters) document what they return.
- tail_wait / tail_abort release the GIL while they block.
No kernel, reduction order, release condition or credit target changes.
Co-Authored-By: Claude Code <noreply@anthropic.com>
…yout tests - Argument checks of GinOverlapPrimitive (missing expert group, bucket sizes not divisible by the group size, per-block numel not a multiple of 4, wrong 2D bucket kind or padded size) raise RuntimeError instead of assert, so python -O keeps them. - The cross-rank consistency check (crc32 + all_reduce MAX per group, then all_reduce MIN) is one module-level _all_ranks_same used by both the recv pool layout and the 2D-bucket table; zlib / statistics are imported at the top. - The module docstring states the engine's lifetime: it lives as long as the process, shutdown_process_group / restart_process_group are not supported, and a failed init cannot be retried. - test/collective/test_gin_overlap_layout.py: CPU tests of _send_layout, _recv_pool_ring (no wrap, wrap at the end of the segment, zero-size slots left out, predecessor distance in 1..n, slot larger than the segment) and _recv_pool_sizes (proportional split, 4 KB rounding, too small pools), and that _BUILD_HINT names every condition of the CMake gate. No GPU and no paddle.base.core.gin_overlap needed. Co-Authored-By: Claude Code <noreply@anthropic.com>
kernels/release.cu has no kernel and no <<<>>>: it is the host-side release state machine and its std::thread. kernels/ holds only the kernel launch wrappers, so the file moves next to gin_engine.cc as release.cc and is compiled by the host compiler instead of nvcc. The content is unchanged apart from the file name in comments. Co-Authored-By: Claude Code <noreply@anthropic.com>
….30.7 gate Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
…fsets in release setup Co-Authored-By: Claude Code <noreply@anthropic.com>
Remove gin_overlap/ from the cpplint exclude list and fix every finding: C-style casts become static_cast / reinterpret_cast, long long and unsigned long long become int64_t / uint64_t, non-const reference out-parameters become pointers (call sites updated), and C system headers come before C++ ones. atomicAdd_system keeps its unsigned long long prototype through one aliased type. No change to values, types seen by Python, or arithmetic. Co-Authored-By: Claude Code <noreply@anthropic.com>
…e slot as an out-parameter C++: both return 0 or a cudaError_t code and write the slot to *slot: cudaErrorNotReady without configure / release_setup, the error of paddle_compute_stream or cudaEventRecord, and cudaErrorLaunchOutOfResources when this step's tail mark slots are used up (was -1 / -2). The Python binding still returns the slot and raises RuntimeError with the error text on failure. CPU test of the binding before setup. Co-Authored-By: Claude Code <noreply@anthropic.com>
…+ sources only The four warning options are wrapped in $<$<COMPILE_LANGUAGE:CXX>:...>, so nvcc no longer receives them for kernels/*.cu. Co-Authored-By: Claude Code <noreply@anthropic.com>
Paddle-Review-Bot
left a comment
There was a problem hiding this comment.
编译选项问题已在 858be8c 解决。仍待处理:test/collective/ 缺少与 reduce_scatter / all_gather 对比的多进程用例。
…edit Co-Authored-By: Claude Code <noreply@anthropic.com>
Paddle-Review-Bot
left a comment
There was a problem hiding this comment.
34efbce 只补充了 launch_signal_credit 的网格大小前置条件说明,两处调用都按 (num_peers + 127) / 128 计算块数,满足该条件。仍待处理:test/collective/ 缺少与 reduce_scatter / all_gather 对比的多进程用例。
PR Category
Distributed Strategy
PR Types
New features
Description
[WIP] do not merge. Opened so the code layout can be reviewed; it is split for review into
#79967 (dynload, kernels, build gate) → #79970 (engine, release layer) → #79971 (pybind, Python layer, packaging, tests); applied in that order they equal this branch. #79966 was merged into #79967, #79968 and #79969 into #79970; those three are closed.
Compiled with the gate on (CUDA 13.2, NCCL 2.31.2,
CUDA_ARCH_BIN="90 100 103"; build log in #79967); not run on a GPU yet. A 4-node bitwise verification is the gate before this leaves WIP.This adds a communication primitive for sharding stage 1: the gradient
reduce_scatterand the parameter
all_gatherof aFusedCommBufferbucket, done with NCCL GIN one-sidedputs instead of collectives. It is the "how to send" half of an optimization that issues
these puts during the backward pass, in the windows where the MoE all-to-all is not using
the network. The "when to send" half (the timing policy, which follows DeepEP) stays in
PaddleFleet and drives this primitive through the gradient-comm backend extension point
(#79958). The two PRs do not depend on each other.
What the primitive does per bucket. The sharding group is split into an intra-node
and an inter-node dimension. Intra-node, each card copies the rows its peers own into
their receive windows over NVLink with the copy engine, then each card sums its own rows;
inter-node, each card puts its partial sum to the other nodes with
ncclGin::putplus asignal, then sums what it received. The result is exactly the shard
reduce_scatterwould have produced; the summation order is fixed, so it is bitwise reproducible.
A smaller receive pool is reused across buckets in a fixed order. Every write waits for a
credit counter showing that the slot's previous occupant has been consumed; every sum
waits for an arrival or signal counter showing that its inputs have landed. The release
layer (
release.cc) holds that state for the whole step: the main threadadvances it synchronously, and a background C++ thread finishes the step after the last
all-to-all.
Layout (follows
deep_ep, which has the same CUDA + pybind + Python shape):paddle/fluid/distributed/collective/gin_overlap/— the engine (windows, bucket tables,submission), the release layer, and the kernels (reduce / put / signal / scale).
Gated in
collective/CMakeLists.txtonWITH_DISTRIBUTE,WITH_GPU,WITH_NCCL, an arch ≥ 90(
ARCH_BIN_CONTAINS_GE_90), CUDA ≥ 13.0 and NCCL ≥ 2.30.7 withnccl_device.hpresent.paddle/fluid/pybind/gin_overlap_api.{cc,h}—paddle.base.core.gin_overlap(
GradCommEngineand the release-layer constants).paddle/phi/backends/dynload/— the two NCCL device-API host entry points(
ncclDevCommCreate,ncclGetLsaDevicePointer) andcuMemcpy2DAsync. Paddle keepsdlopen-ing NCCL; the device side is header-inlined and adds no link dependency.
python/paddle/distributed/communication/gin_overlap/— the window and receive-poollayout, computed in Python and cross-checked across ranks. Not exported from any parent
package.
test/collective/test_gin_overlap_layout.py— CPU tests of the receive-pool layout helpers.The code was moved from PaddleFleet (where earlier states were bitwise-verified against Paddle's
reduce_scatteron 4 nodes / 32 cards, AdamW and Muon) with changes limited to includepaths, namespaces and the build wiring.
Changes after the first review of the split PRs (details in #79967 / #79970 / #79971):
comments, docstrings, errors and logs in English (own commit; outside string literals the token
stream / AST is identical); the dynload wrappers join the existing groups; the gate requires
CUDA ≥ 13.0 (the
cudaMemcpyBatchAsyncsignature); bounds checks without wrap-around andnumel < 0checks;configure/set_comms/release::setuproll back on failure, the otherinit steps document that a failure needs a restart; the engine's lifetime is the process
(
shutdown_process_group/restart_process_groupunsupported);release::setup/setup_gather_tablevalidate every offset; a duplicatebucket_readyis caught in RUN too;tail_wait/tail_abortrelease the GIL; the release layer moved torelease.cc; Pythonargument checks raise instead of assert. No kernel body, summation order, release condition,
credit target or coefficient placement changed; none of these changes has run on a GPU.
Changes after the second review (details in #79967 / #79970 / #79971):
release::setuprequires the densesend_offsetandrecv_offset_intrato be 16-byte aligned (k_reduce_intra_nodereads them asfloat4*), otherwisecudaErrorInvalidValue;gin_overlap/is no longer excluded from cpplint and every finding is fixed in code (C++ casts,int64_t/uint64_t, pointer out-parameters, include order; oneNOLINT(runtime/int)foratomicAdd_system'sunsigned long longprototype); comments: the dynload group of the device-API host entry points, how libcudart is linked (dynamically,--cudart shared), the return value ofrecord_window_mark. SASS check:cuobjdump -sassof the fourkernels/*.cu.obefore and after the change, per kernel and arch (sm_90 / sm_100 / sm_103): reduce 9/9, put 12/12, signal 15/15, scale 3/3 functions with identical instruction bodies (reduce / put / scale byte-identical; in signal only the mangled names ofk_store_release/k_signal_intra_arrivechange,y->m, fromunsigned long long->uint64_t). Incremental build of exactly this tree (no re-cmake,make -j180, 18:18-19:16, exit 0, noerror:in the log):Built target gin_overlap,libpaddle.solinks (its symbols carry the pointer signatures), the wheel containspaddle/distributed/communication/gin_overlap/.Follow-up of the second review:
record_window_markandtail_markreturn an error code with the slot as an out-parameter (each failure has its owncudaError_tcode); the Python binding still returns the slot and raises RuntimeError with the error text. Incremental build of this tree (no re-cmake): only gin_engine.cc, paddle_tensor.cc, release.cc and pybind (gin_overlap_api.cc, plus the generated eager_op_function.cc) recompiled; the four kernels/*.cu.o are unchanged (same md5); libgin_overlap.a and libpaddle.so link, the wheel builds. test_gin_overlap_layout.py passes on that build (15 cases, including the two binding cases). Callers: no C++ or Python caller in Paddle other than the binding; PaddleFleet's scheduler calls record_window_mark() / tail_mark() through the binding and gets the slot as before (its < 0 checks only apply to older builds).The multi-process bitwise test against NCCL
reduce_scatter/all_gatherin Paddle is added after the Paddle side is merged; the multi-card numerical check is the 8-card PaddleFleet test (PaddleFleet #2577, GIN vs NCCL, bitwise).The four
-Wno-*options oftarget_compile_options(gin_overlap ...)now apply to C++ sources only (each wrapped in$<$<COMPILE_LANGUAGE:CXX>:...>, P0 of the first review): inflags.makeof the build,CXX_FLAGShas them andCUDA_FLAGSdoes not; incompile_commands.jsonthe four.ccfiles carry them and the fourkernels/*.cudo not. The rebuild (cmake re-run by make, 0error:lines, no warning from gin_overlap files,libgin_overlap.aandlibpaddle.solink, wheel built) recompiled the kernels;cuobjdump -sassbefore / after is identical per kernel and arch (reduce 9/9, put 12/12, signal 15/15, scale 3/3 functions, SASS text byte-identical).Not verified: the precision acceptance run (4 nodes, bitwise) is still in progress, and the changes of this round have not been through it.
是否引起精度变化
否
🤖 Generated with Claude Code