Repository navigation
Conversation
Paddle-Bot Review Board (review完成)
可维护性建议(总分 84.8 / 100,只给建议,不影响评审结论)
各维度得分(仅评本 PR 改动,点击展开)
Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully. |
| // dense 组用于机内到达格、机内 credit 和 2D gather 的同节点 CE;专家组只用于 | ||
| // 2D gather | ||
| void** lsa_pointers_device = nullptr; | ||
| CUDA_CHECK(cudaMalloc((void**)&lsa_pointers_device, 16 * sizeof(void*))); |
There was a problem hiding this comment.
configure 失败时不释放已经创建的资源,重试也不安全。
- 234-259 行已经写入 credit 偏移,创建了 4 条侧流和全部 event;264 行又
cudaMalloc了lsa_pointers_device。之后 271/281/287/291/295/299 行任何一处失败都直接return:device 缓冲泄漏,侧流和 event 留在全局变量里。 - 此时
g_buckets_ready仍为 0,再调一次configure会重新cudaStreamCreateWithFlags/cudaEventCreateWithFlags并覆盖旧句柄,旧的流和 event 再也释放不了。 - 其它初始化步骤也有同样的问题:
create_dev_comm在ncclDevCommCreate成功之后才检查lsaSize(180 行),检查失败时 devcomm 不销毁。alloc_windows中途失败后comm_group.send已经非空,重试直接返回cudaErrorNotReady。set_comms里 dense 组成功、专家组失败时,dense 组的 comm 不会清掉,重试只会报"已经设置过了"。
建议:configure 先在局部变量里建好资源,或者用统一的清理出口,失败时 cudaFree / cudaStreamDestroy / cudaEventDestroy 并把全局变量复位。其它步骤要么在失败时回滚本步写入的状态,要么在注释和 Python 侧写明"失败后不可重试,必须重启进程"。
| if (bytes < 4096) bytes = 4096; | ||
| NCCL_CHECK(ncclMemAlloc((void**)out_ptr, bytes)); | ||
| CUDA_CHECK(cudaMemset(*out_ptr, 0, bytes)); | ||
| NCCL_CHECK(ncclCommWindowRegister( |
There was a problem hiding this comment.
window 直接注册在 Paddle 的 comm 上,但没有任何反注册或释放路径。Paddle 侧 shutdown / restart 之后,这些状态会悬空。
- 这里和 366-371 行用
ncclCommWindowRegister直接注册到paddle_nccl_comm取回的ncclComm_t上。NCCLCommContext::DestroyNCCLComm只反注册经NCCLCommContext::RegisterWindow登记进windows_的窗口(nccl_comm_context.cc:104-110、169-179),这几块 window 不在其中。 ProcessGroupNCCL::Shutdown/Restart(process_group_nccl.cc:1020-1045)会销毁或重建这个 comm。之后g_groups[].comm、dev_comm、4 个 window 和lsa_*指针都指向已销毁的对象,而g_buckets_ready仍为 1。- 整个系列都没有调用
ncclCommWindowDeregister、ncclDevCommDestroy、ncclMemFree、cuMemUnmap/cuMemRelease或cudaStreamDestroy。send/recv/signal window 和 HOST_NUMA 内存只能随进程退出回收。
建议二选一:
- 提供一个释放接口,在 comm 销毁前按逆序执行 Deregister →
ncclDevCommDestroy→ncclMemFree/ unmap / release → 销毁流和 event,再复位g_groups和g_buckets_ready。文档里写明它必须在shutdown_process_group/restart_process_group之前调用。 - 改为走
NCCLCommContext::RegisterWindow,让 Paddle 销毁 comm 时一并反注册。
| const char* what) { | ||
| if (!t.defined() || !t.initialized() || !t.is_dense_tensor() || | ||
| t.place().GetType() != phi::AllocationType::GPU) { | ||
| fail(std::string("fused all_gather 的 2D 拷贝:") + what + |
There was a problem hiding this comment.
💡 可维护性建议(AI 生成债务,不阻塞合入):69/73/80/177 行的错误信息都以"fused all_gather 的 2D 拷贝"开头,193 行写的是"fused all_gather 拆回"。但 copy_2d 是通用的 pitched 拷贝,这个系列里没有叫 fused all_gather 的调用方,用户排查时会被引向一个不存在的功能。建议统一改成 copy_2d: 前缀,和 send_window_tensor: 的写法保持一致。
| &comm_group.send, | ||
| &comm_group.win_send))) | ||
| return rc; | ||
| comm_group.send_bytes = send_bytes < 4096 ? 4096 : send_bytes; |
There was a problem hiding this comment.
💡 可维护性建议(逻辑清晰度,不阻塞合入):"最少 4096 字节"这条规则在 alloc_symmetric_window(43 行)和这里各写了一遍。规则一改,两处容易不同步。另外这里只记录了 send window 的实际大小,recv / signal 的实际大小没有保存。建议让 alloc_symmetric_window 通过出参返回实际分配的字节数,调用方直接用它填 send_bytes(需要时也记下 recv / signal 的大小),去掉这里的重复计算。
| #x, \ | ||
| (int)_res, \ | ||
| _msg ? _msg : "?"); \ | ||
| return (int)cudaErrorUnknown; \ |
There was a problem hiding this comment.
💡 可维护性建议(错误处理一致性,不阻塞合入):返回码约定是"<10000 为 cudaError_t,>=10000 为 10000 + ncclResult_t"。CUDA_CHECK 和 NCCL_CHECK 都保留了原始错误码,只有 CUDA_DRIVER_CHECK 把所有驱动错误都压成 cudaErrorUnknown,具体的 CUresult 只打印到 stderr。这样 Python 侧经 error_string 只能拿到 "unknown error",在多进程日志里也很难和 stderr 对上。建议在约定里给 CUresult 单独分一段(例如 20000 + CUresult),并在 error_string 里用 cuGetErrorString 还原。
| size_t send_offset, | ||
| long long numel); // NOLINT | ||
|
|
||
| void copy_2d(const paddle::Tensor& dst, |
There was a problem hiding this comment.
💡 可维护性建议(测试策略,不阻塞合入):本层新加的 host 侧代码里有不少参数校验逻辑,但目前没有任何测试入口,PR 描述也说明没有测试调用引擎。copy_2d 只依赖普通 GPU 张量和计算流,不需要 NCCL window,可以先给它加一个单卡测试(在门控开启时编译),覆盖越界、width > pitch、非连续张量这几条报错路径,再验证一次正常拷贝的结果。
| " / pitch " + std::to_string(pitch) + " B 不合法"); | ||
| } | ||
| const size_t nbytes = (size_t)t.numel() * phi::SizeOf(t.dtype()); | ||
| const size_t need = offset + (height - 1) * pitch + width; |
There was a problem hiding this comment.
checked_block 把 numel × 元素大小 当作可访问的字节数,而且越界计算可能回绕,因此拦不住 Python 传入的越界参数。
copy_2d会经 pybind 直接暴露给 Python([WIP][Distributed] gin_overlap 3/3: pybind, Python layer, packaging and tests #79971),参数都来自外部。- Paddle 默认开启 stride kernel(
FLAGS_use_stride_kernel=true)。expand这类视图的 stride 为 0,numel × 元素大小大于实际分配:这里的检查能通过,但cuMemcpy2DAsync按连续内存读写,会越界访问。转置等非连续视图也会让字节偏移和张量元素对不上。 - 78 行的
offset + (height - 1) * pitch + width在height、pitch很大时会回绕成小值,越界检查随之失效。
建议:
- 增加
t.is_contiguous()检查。 - 把越界判断改成不会回绕的写法:先确认
offset <= nbytes && width <= nbytes - offset,再判断height - 1 <= (nbytes - offset - width) / pitch。72 行已经保证pitch >= width > 0,这里的除法是安全的。
| extern CommGroup g_groups[NUM_GROUPS]; | ||
| inline CommGroup& group_at(int group) { return g_groups[group]; } | ||
| inline CommGroup& dense_group() { return g_groups[GROUP_DENSE]; } | ||
| inline CommGroup& expert_group() { return g_groups[GROUP_MOE]; } |
There was a problem hiding this comment.
💡 可维护性建议(命名质量,不阻塞合入):同一个组用了两套名字:GROUP_MOE、num_moe_buckets、g_event_moe_* 和 credit_moe_offset 用 moe;expert_group() 以及 set_comms 的 expert_ring_id / expert_size / expert_rank 用 expert。读代码的人得自己确认这两套名字指的是同一个组。建议统一成一个词。
173a161 to
03eaeb5
Compare
Paddle links neither libnccl nor libcuda, so new code that calls them has to go through the phi::dynload dlopen wrappers. The gin_overlap gradient communication added in the following commits needs three symbols that are not wrapped yet: * ncclDevCommCreate / ncclGetLsaDevicePointer, the two host entry points of NCCL's device (LSA/GIN) API. They are declared in nccl_device.h, which also carries the device-side inline implementations of the whole device API, so including it from nccl.h would land all of that in every translation unit that reaches for any NCCL symbol. The declarations it needs are repeated here instead, the way the internal entry points at the top of the file already are, and are skipped in translation units that include nccl_device.h themselves. Same 2.30.7 floor as the zero-SM group above them: below that the feature is refused, not degraded. * cuMemcpy2DAsync, for the pitched copy that splits a fused all_gather back into per-bucket storages. It is wrapped under the name libcuda exports, cuMemcpy2DAsync_v2: the wrapper dlsym()s the name it is given but types itself from decltype(&::name), which cuda.h's macro resolves to the v2 declaration, so passing the plain name would call the v1 entry point (whose parameter is the old CUDA_MEMCPY2D_v1 layout) through a v2 signature. Call sites keep writing cuMemcpy2DAsync and let cuda.h's macro rewrite it. The host overload of ncclTeamWorld is deliberately not wrapped: it shares a name with the device-side ncclTeamWorld(ncclDevComm const&), so decltype(&::ncclTeamWorld) cannot be formed, and no host code needs it. Co-Authored-By: Claude Code <noreply@anthropic.com>
ncclDevCommCreate and ncclGetLsaDevicePointer need the same NCCL 2.30.7 as the other symbols of NCCL_RAND_ROUTINE_EACH_AFTER_23007, so they join that group instead of a separate NCCL_RAND_ROUTINE_EACH_DEVICE_API under an identical #if. cuMemcpy2DAsync_v2 has no version gate and joins CUDA_ROUTINE_EACH; CUDA_ROUTINE_EACH_MEMCPY and its two expansions go. The comment on the nccl_device.h prototypes now states what actually holds: the prototypes must match nccl_device.h exactly, because this header is usually included first and both declarations then coexist; the guard only drops the duplicate in the other include order. Co-Authored-By: Claude Code <noreply@anthropic.com>
….30.7 gate Co-Authored-By: Claude Code <noreply@anthropic.com>
03eaeb5 to
86cad1d
Compare
86cad1d to
46ff318
Compare
First piece of moving the NCCL-GIN sharding-stage1 gradient primitives out of PaddleFleet's extension (paddlefleet_ops/gin_overlap/csrc) into Paddle: the CUDA kernels and their launch wrappers, plus the CMake gate that decides whether the directory is built at all. Nothing calls these yet; the host-side engine that launches them comes in the next commits. * kernels/reduce.cu: the intra-node column sum, the dense shard sum across nodes and the expert shard sum, each scaled in place. * kernels/put.cu: GIN one-sided puts to the same-rank peers on other nodes, to the expert-group peers, to the 2D-bucket owner, and the GIN flush. * kernels/signal.cu: the intra-node counter / arrival writes over LSA pointers, the GIN credit signal, a single release store, and the lookup of the host-window LSA pointers. * kernels/scale.cu: an in-place scale of a send-window range. * kernels/kernels.h: the launch_* / preload_* declarations the host side is allowed to call (no <<<>>> outside kernels/). * gin_dynload.h: brings the phi::dynload wrappers (previous commit) into namespace gin_overlap under their plain names, so the host code that comes next calls NCCL / the driver through dlopen without touching call sites. The code is moved, not rewritten: the sum order, the kernels and the block counts are untouched; the code-level diff against the extension is include paths, namespaces and include guards (#pragma once). Two casts of CUDA built-ins are respelled as static_cast so that the no-int64-type-cuda-index-without-cast ast-grep rule sees them; cpplint is switched off for the directory, as it is for deep_ep. The comments say what each kernel does and what must hold; reduce.cu's header lists what keeps the result bit-identical to reduce_scatter (ring order starting at own rank + 1, __fmul_rn before __fadd_rn) and states that the block / thread count is not among them: it only decides which indices a thread walks. The caller does change the reduce kernels' block count between windows within one run, and the bit-exactness check against reduce_scatter passes with that. Build gate (CMake is written but not run in this change): * arch: only built when NVCC_ARCH_BIN contains 9.0 or newer, computed the way paddle/phi/kernels/CMakeLists.txt computes has_arch_ge80. Inside the directory the <= 8.x gencodes are stripped off CMAKE_CUDA_FLAGS, as deep_ep/CMakeLists.txt does. * NCCL >= 2.30.7 with nccl_device.h present, and CUDA >= 12.8 for cudaMemcpyBatchAsync. The result is cached as WITH_GIN_OVERLAP for the pybind layer added later. gin_dynload.h #errors below the NCCL floor as a backstop for the gate. Co-Authored-By: Claude Code <noreply@anthropic.com>
Translate every comment, docstring, error message and log line of the gin_overlap sources (C++, CUDA, CMake, pybind and the Python layer) into English, and tighten the wording on the way. The terms are defined once, in the gin_engine.h file header (C++) and in the engine.py module docstring (Python); every other file uses them as defined there. Strings only. comment_clean_check.py --ignore-strings passes on every changed file: C++ / CUDA / CMake token streams are identical except for string literals (printf conversions inside merged literals keep their order), and the Python ASTs are identical except for str constants and the literal parts of f-strings. Log lines all carry the [gin_overlap] prefix; copy_2d errors now start with "copy_2d:". Co-Authored-By: Claude Code <noreply@anthropic.com>
…the gate Both cudaMemcpyBatchAsync calls use the CUDA 13 signature (8 arguments, const void** sources). CUDA 12.8 / 12.9 declare a 9-argument version, so a build with the gate open on those toolkits would not compile. Raise the gate to CUDA >= 13.0, the only toolkit the code has been built and run with, instead of adding an untested 12.x branch; gin_dynload.h gets the matching #error backstop. The batch attribute index is named attrs_idx, which is what that argument is. The arch condition reuses ARCH_BIN_CONTAINS_GE_90, computed at the top of the same file for DeepEP, instead of a second loop over NVCC_ARCH_BIN. _BUILD_HINT in the Python layer now lists exactly the gate's conditions, and the CMake comment says to change both together. Co-Authored-By: Claude Code <noreply@anthropic.com>
…ne API
Argument checks (values come from Python):
- scale_range, gather_put_to_owner, send_window_tensor and copy_2d check
their bounds without unsigned wrap-around, and reject numel < 0.
copy_2d also rejects non-contiguous tensors (a stride-0 or transposed
view passes a numel-based bound but the copy reads contiguous bytes).
- release::setup checks every offset of the per-bucket tables and the
host window areas: non-negative, aligned (counters 8 B, data 4 B) and
inside its window. setup_gather_table checks both arrival counter
offsets of each entry the same way. CommGroup records recv_bytes for
this, and alloc_symmetric_window returns the size it actually
registered instead of the caller repeating the 4096 B minimum.
- A wrong state returns cudaErrorNotReady and a bad argument
cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
- bucket_ready rejects a bucket reported twice in a step in the RUN state
too, from a bitmap only the main thread touches.
Init failures:
- configure destroys the streams and events it created, frees its
temporary buffer and resets the globals it set when a later step fails,
so it can be retried. set_comms clears the groups it set on failure.
release::setup unpublishes its control block if the tail thread cannot
be started. The steps that cannot be rolled back (window registration,
devcomm creation, the host window) say so: restart the process.
- The gin_engine.h constraints state that the engine lives as long as
the process; shutdown_process_group / restart_process_group are not
supported.
API:
- GIN_CHECK replaces the dozen { rc = launch_x(...); if (rc) return rc; }
blocks. launch_* / preload_* return cudaError_t.
- launch_put_to_expert_peers takes (send_offset, recv_offset) like
launch_put_to_node_peers.
- join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert),
as bucket_ready and the gather table do, instead of is_dense.
- The six release positions are named variables, one per stage.
- The methods that do not return a code (record_window_mark, tail_mark,
bucket_ready, tail_all_posted, tail_wait, tail_abort, release_activity,
the version getters) document what they return.
- tail_wait / tail_abort release the GIL while they block.
No kernel, reduction order, release condition or credit target changes.
Co-Authored-By: Claude Code <noreply@anthropic.com>
kernels/release.cu has no kernel and no <<<>>>: it is the host-side release state machine and its std::thread. kernels/ holds only the kernel launch wrappers, so the file moves next to gin_engine.cc as release.cc and is compiled by the host compiler instead of nvcc. The content is unchanged apart from the file name in comments. Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Remove gin_overlap/ from the cpplint exclude list and fix every finding: C-style casts become static_cast / reinterpret_cast, long long and unsigned long long become int64_t / uint64_t, non-const reference out-parameters become pointers (call sites updated), and C system headers come before C++ ones. atomicAdd_system keeps its unsigned long long prototype through one aliased type. No change to values, types seen by Python, or arithmetic. Co-Authored-By: Claude Code <noreply@anthropic.com>
…+ sources only The four warning options are wrapped in $<$<COMPILE_LANGUAGE:CXX>:...>, so nvcc no longer receives them for kernels/*.cu. Co-Authored-By: Claude Code <noreply@anthropic.com>
…treams Second piece of the move from PaddleFleet's extension: the host side that builds everything the per-step code runs on. Nothing calls it yet (the Python binding comes last), and nothing is submitted to the GPU per step yet (next commit). * gin_engine.h / gin_engine.cc, initialization only: the CommGroup state of the two communication groups (sharding / moe_sharding), the error-check macros, and the GradCommEngine methods that the Python layer calls once at startup, in this order: set_comms (reuse Paddle's ncclComm_t for each ring id and cross-check size / rank), alloc_windows (send / recv / signal as NCCL_WIN_COLL_SYMMETRIC windows), alloc_host_counters (one HOST_NUMA host window per group, registered with NCCL_WIN_STRICT_ORDERING), create_dev_comm (ncclDevCommCreate + LSA pointers), set_group_topology, and configure (side streams, per-bucket events, the window-mark ring, the credit offsets, the host-window LSA pointers, and preloading every kernel so that the first launch of a module does not land on the backward critical path). Also error_string and the NCCL runtime / build version queries. * paddle_comm.cc: takes the ncclComm_t out of Paddle's ProcessGroup by ring id, the way DeepEP does (ProcessGroupMapFromGid -> ProcessGroupNCCL -> NCCLCommContext::GetNcclComm). * paddle_tensor.cc / .h: Paddle's current compute stream, the zero-copy paddle::Tensor over a send-window slice (paddle::from_blob with deleter = nullptr, so the memory stays owned by the NCCL window, plus a post-wrap self-check), and the pitched copy_2d through cuMemcpy2DAsync. The file header of gin_engine.h is the one full description of the C++ side (terms, the two groups, the dense / expert / 2D stages, the receive pools, the host-window layout and the constraints). It is added here in full: it is organised by concept, not by layer, so the parts about the per-stage submits and the release layer (next two commits) cannot be moved out without rewriting it. paddle_comm.cu and paddle_tensor.cu become .cc: they were .cu only because an out-of-tree build has to hand any translation unit that includes Paddle headers to nvcc (thrust); in-tree they are ordinary sources. The pybind registration that was stranded in paddle_tensor.cu moves to the pybind layer, so paddle_tensor.h declares the real paddle::Tensor signatures. The library now also depends on process_group_nccl, for paddle_comm.cc. Like the kernels, the code is moved, not rewritten: same checks, same order of collective calls. Co-Authored-By: Claude Code <noreply@anthropic.com>
Translate every comment, docstring, error message and log line of the gin_overlap sources (C++, CUDA, CMake, pybind and the Python layer) into English, and tighten the wording on the way. The terms are defined once, in the gin_engine.h file header (C++) and in the engine.py module docstring (Python); every other file uses them as defined there. Strings only. comment_clean_check.py --ignore-strings passes on every changed file: C++ / CUDA / CMake token streams are identical except for string literals (printf conversions inside merged literals keep their order), and the Python ASTs are identical except for str constants and the literal parts of f-strings. Log lines all carry the [gin_overlap] prefix; copy_2d errors now start with "copy_2d:". Co-Authored-By: Claude Code <noreply@anthropic.com>
…ne API
Argument checks (values come from Python):
- scale_range, gather_put_to_owner, send_window_tensor and copy_2d check
their bounds without unsigned wrap-around, and reject numel < 0.
copy_2d also rejects non-contiguous tensors (a stride-0 or transposed
view passes a numel-based bound but the copy reads contiguous bytes).
- release::setup checks every offset of the per-bucket tables and the
host window areas: non-negative, aligned (counters 8 B, data 4 B) and
inside its window. setup_gather_table checks both arrival counter
offsets of each entry the same way. CommGroup records recv_bytes for
this, and alloc_symmetric_window returns the size it actually
registered instead of the caller repeating the 4096 B minimum.
- A wrong state returns cudaErrorNotReady and a bad argument
cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
- bucket_ready rejects a bucket reported twice in a step in the RUN state
too, from a bitmap only the main thread touches.
Init failures:
- configure destroys the streams and events it created, frees its
temporary buffer and resets the globals it set when a later step fails,
so it can be retried. set_comms clears the groups it set on failure.
release::setup unpublishes its control block if the tail thread cannot
be started. The steps that cannot be rolled back (window registration,
devcomm creation, the host window) say so: restart the process.
- The gin_engine.h constraints state that the engine lives as long as
the process; shutdown_process_group / restart_process_group are not
supported.
API:
- GIN_CHECK replaces the dozen { rc = launch_x(...); if (rc) return rc; }
blocks. launch_* / preload_* return cudaError_t.
- launch_put_to_expert_peers takes (send_offset, recv_offset) like
launch_put_to_node_peers.
- join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert),
as bucket_ready and the gather table do, instead of is_dense.
- The six release positions are named variables, one per stage.
- The methods that do not return a code (record_window_mark, tail_mark,
bucket_ready, tail_all_posted, tail_wait, tail_abort, release_activity,
the version getters) document what they return.
- tail_wait / tail_abort release the GIL while they block.
No kernel, reduction order, release condition or credit target changes.
Co-Authored-By: Claude Code <noreply@anthropic.com>
kernels/release.cu has no kernel and no <<<>>>: it is the host-side release state machine and its std::thread. kernels/ holds only the kernel launch wrappers, so the file moves next to gin_engine.cc as release.cc and is compiled by the host compiler instead of nvcc. The content is unchanged apart from the file name in comments. Co-Authored-By: Claude Code <noreply@anthropic.com>
Remove gin_overlap/ from the cpplint exclude list and fix every finding: C-style casts become static_cast / reinterpret_cast, long long and unsigned long long become int64_t / uint64_t, non-const reference out-parameters become pointers (call sites updated), and C system headers come before C++ ones. atomicAdd_system keeps its unsigned long long prototype through one aliased type. No change to values, types seen by Python, or arithmetic. Co-Authored-By: Claude Code <noreply@anthropic.com>
…e slot as an out-parameter C++: both return 0 or a cudaError_t code and write the slot to *slot: cudaErrorNotReady without configure / release_setup, the error of paddle_compute_stream or cudaEventRecord, and cudaErrorLaunchOutOfResources when this step's tail mark slots are used up (was -1 / -2). The Python binding still returns the slot and raises RuntimeError with the error text on failure. CPU test of the binding before setup. Co-Authored-By: Claude Code <noreply@anthropic.com>
46ff318 to
feabde3
Compare
PR Category
Distributed Strategy
PR Types
New features
Description
Merged into #79970.
This PR is stacked on #79967: Files changed also shows the earlier layers. Changes of this layer only: AlAuAu/Paddle@gin_split_2_kernels...gin_split_3_engine_init
Part 3/6 of the gin_overlap primitive (#79964), split for review. Compiled with the gate on (CUDA 13.2, NCCL 2.31.2,
CUDA_ARCH_BIN="90 100 103"); not run on a GPU yet.The six PRs of this series: #79966 → #79967 → #79968 → #79969 → #79970 → #79971. Each branch contains the ones before it, so on GitHub every PR after the first shows the earlier layers too; review only the commits of this layer (the Commits tab), and merge in order.
Series (each PR builds on the previous one; the six together equal #79964):
1 dynload wrappers → 2 kernels + build gate → 3 engine setup (this PR) → 4 per-stage submits →
5 release layer → 6 pybind + Python layer.
What this PR adds
The host side that builds everything the per-step code runs on. Nothing is submitted to the GPU
per step yet (#79969).
gin_engine.h: the file header is the one full description of the C++ side: terms (bucket,shard, GIN, LSA, devcomm, CE, window, mark, release, credit, owner, tail thread), the two groups,
the four dense stages / two expert stages / Muon 2D gather, the receive pools, the host-window
layout, and the constraints every later file relies on. It is added here in full. It is
organised by concept, not by PR, and the parts about the release layer ([WIP][Distributed] gin_overlap 2/3: engine and release layer (C++) #79970) are single
bullets or single sentences inside it (e.g. "the release layer reads remote counters on the
CPU", "only one writer of the release state: advance or the tail thread"); cutting those out
would mean rewriting the description, so it is not split.
gin_engine.h/gin_engine.cc, initialization only:CommGroupstate of the two groups(sharding / moe_sharding), the
CUDA_CHECK/NCCL_CHECK/CUDA_DRIVER_CHECKmacros, and theGradCommEnginemethods called once at startup, in this order:set_comms(reuse Paddle'sncclComm_tper ring id, cross-check size / rank),alloc_windows(send / recv / signal asNCCL_WIN_COLL_SYMMETRIC),alloc_host_counters(one HOST_NUMA host window per group,registered with
NCCL_WIN_STRICT_ORDERING),create_dev_comm(ncclDevCommCreate+ LSApointers),
set_group_topology,configure(side streams, per-bucket events, the window-markring, credit offsets, host-window LSA pointers, preloading every kernel module). Also
error_stringand the NCCL runtime / build version queries.paddle_comm.cc: thencclComm_tof Paddle'sProcessGroupby ring id, the way DeepEP gets it(
ProcessGroupMapFromGid→ProcessGroupNCCL→NCCLCommContext::GetNcclComm).paddle_tensor.cc/.h: Paddle's current compute stream, the zero-copypaddle::Tensorover asend-window slice (
paddle::from_blob,deleter = nullptr, so the NCCL window keeps owning thememory), and the pitched
copy_2dthroughcuMemcpy2DAsync.gin_overlap/CMakeLists.txt: the three.ccfiles joinGIN_OVERLAP_SRCS;DEPSgainsprocess_group_nccl(forpaddle_comm.cc).Comments in this PR mention methods and files that arrive later (
release_setup, the pybind file,the Python layer); they are kept as they are in #79964.
Depends on: #79967 (
gin_split_2_kernels).Callers after merge: none. The methods are called from Python, which is bound in #79971.
What CI can show: on a build with the gate on (see #79967), that these files compile against
Paddle's headers and that the declarations in
gin_engine.hmatch the definitions (a staticcheck of declarations, includes and
GIN_OVERLAP_SRCSpasses for this branch). No test callsthe engine.
How correctness is established: moved from PaddleFleet's extension, same checks and same order
of collective calls;
paddle_comm.cu/paddle_tensor.cubecame.cc(in-tree they need nonvcc). The bitwise evidence so far is for earlier, out-of-tree states of the engine (4 nodes /
32 cards, 10 steps, identical to Paddle's
reduce_scatterbaseline; see #79967), not for this tree.The acceptance gate is the same 10-step AdamW and Muon run on a build of the whole series.
The whole series has been compiled (see #79967); nothing has been run on a GPU yet.
Changes after the first review
copy_2drejects non-contiguous tensors and checks its bounds without unsigned wrap-around; its errors start withcopy_2d:.send_window_tensorchecks its bounds the same way.configuredestroys the streams and events it created, frees its temporary buffer and resets the globals it set, so it can be retried. A failedset_commsclears the groups it set. The steps that cannot be rolled back (alloc_windows,alloc_host_counters,create_dev_comm: registration and devcomm creation are collective) say in their comments that a failure needs a process restart.gin_engine.hheader and the Python docstring state that the engine lives as long as the process and thatshutdown_process_group/restart_process_groupare not supported (using the engine after them is undefined). No Python-side interception was added:restart_process_grouprebuilds the comm inside the same group object (same.id), so a cheap Python check cannot see it, and a reliable one needs a hook inProcessGroupNCCL.alloc_symmetric_windowreturns the size it registered;CommGrouprecordsrecv_bytestoo.GIN_CHECKreplaces the{ rc = launch_x(...); if (rc) return rc; }blocks. The methods that do not return a code in the usual convention say what they return.CUresultinCUDA_DRIVER_CHECK, and one name formoe/expert(renames across the whole series). A single-cardcopy_2dtest needs a GPU; not added.Not verified: none of these changes has run on a GPU. Every change is outside the numerical path (no kernel body, summation order,
__fmul_rn/__fadd_rnorder, release condition, credit target or coefficient placement changed); the 4-node bitwise acceptance run is still to come. Comments, docstrings, errors and logs of the series are now in English (one commit per layer; outside string literals the token stream / AST is identical, checked mechanically).Changes after the second review
int64_t/uint64_t, pointer out-parameters, include order); details in [WIP][Distributed] gin_overlap 1/3: dynload wrappers, device kernels and the build gate #79967.record_window_mark/tail_markamong the methods that return a slot (they return an error code now, see [WIP][Distributed] gin_overlap 4/6: per-stage submit functions and the 2D-bucket gather #79969 / [WIP][Distributed] gin_overlap 2/3: engine and release layer (C++) #79970).Not verified: the precision acceptance run (4 nodes, bitwise) is still in progress, and the changes of this round have not been through it. The behavior changes of this round are the argument check in
release::setup(#79970) and the error codes ofrecord_window_mark/tail_mark(#79969 / #79970, Python still gets the slot); the cpplint fixes (#79967) change casts, integer type spellings and reference -> pointer parameters only.是否引起精度变化
否
🤖 Generated with Claude Code