Skip to content

[WIP][Distributed] gin_overlap 4/6: per-stage submit functions and the 2D-bucket gather - #79969

Closed
AlAuAu wants to merge 25 commits into
PaddlePaddle:developfrom
AlAuAu:gin_split_4_engine_submit
Closed

AlAuAu wants to merge 25 commits into
PaddlePaddle:developfrom
AlAuAu:gin_split_4_engine_submit

Conversation

@AlAuAu

@AlAuAu AlAuAu commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

PR Category

Distributed Strategy

PR Types

New features

Description

Merged into #79970.

This PR is stacked on #79968: Files changed also shows the earlier layers. Changes of this layer only: AlAuAu/Paddle@gin_split_3_engine_init...gin_split_4_engine_submit

Part 4/6 of the gin_overlap primitive (#79964), split for review. Compiled with the gate on (CUDA 13.2, NCCL 2.31.2, CUDA_ARCH_BIN="90 100 103"); not run on a GPU yet.

The six PRs of this series: #79966 → #79967 → #79968 → #79969 → #79970 → #79971. Each branch contains the ones before it, so on GitHub every PR after the first shows the earlier layers too; review only the commits of this layer (the Commits tab), and merge in order.

Series (each PR builds on the previous one; the six together equal #79964):
1 dynload wrappers → 2 kernels + build gate → 3 engine setup → 4 per-stage submits (this PR) →
5 release layer → 6 pybind + Python layer.

What this PR adds (gin_engine.h / gin_engine.cc only)

The host functions that hand one stage of one bucket to its side stream, and the stream / event
plumbing around them.

  • The six stage submits: submit_dense_intra_node_copy (cudaMemcpyBatchAsync of the column to the
    other ranks on the node + the arrival write), submit_dense_intra_node_reduce (+ intra-node
    credit), submit_dense_inter_node_put, submit_moe_put, submit_dense_inter_node_reduce and
    submit_moe_reduce (+ GIN credit). Each waits only on work this rank has already handed out (a
    compute-stream mark or a local side-stream event), never on a remote counter.
  • record_window_mark: records a mark on Paddle's current compute stream into the ring
    configure() created; GIN_COMPUTE_STREAM, the macro that fetches that stream.
  • join_bucket_streams (compute stream waits for a bucket's side-stream events), scale_range,
    end_step (GIN flush of both groups so the send window can be reused), stats.
  • Muon 2D-bucket gather to the owner: gather_setup, gather_gate, gather_wait_bucket,
    gather_mark_ready, gather_put_to_owner (GIN put across nodes, CE + release store within a
    node), gather_join. The table-driven entry point gather_put_entry forwards to the release
    layer and comes with it in [WIP][Distributed] gin_overlap 2/3: engine and release layer (C++) #79970.

Depends on: #79968 (gin_split_3_engine_init).

Callers after merge: none. The only caller of submit_* is the release layer (#79970); the
GradCommEngine methods are called from Python (#79971).

What CI can show: on a build with the gate on (see #79967), that the two files compile. No test
calls them.

How correctness is established: moved from PaddleFleet's extension unchanged. The bitwise
evidence so far is for earlier, out-of-tree states of the engine (4 nodes / 32 cards, 10 steps,
identical to Paddle's reduce_scatter baseline; see #79967), not for this tree. The acceptance
gate is the same 10-step AdamW and Muon run on a build of the whole series. The whole
series has been compiled (see #79967); nothing has been run on a GPU yet.

Changes after the first review

  • P1: see [WIP][Distributed] gin_overlap 1/3: dynload wrappers, device kernels and the build gate #79967 (the gate now requires CUDA 13.0); the batch attribute index is named attrs_idx.
  • P2: scale_range and gather_put_to_owner check their bounds without wrap-around and reject numel < 0.
  • A wrong state returns cudaErrorNotReady and a bad argument cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
  • join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert), as bucket_ready and the gather table do, instead of is_dense.
  • (Superseded below: record_window_mark now returns an error code.) record_window_mark keeps returning the slot; its declaration now says so (an out-parameter would change the Python binding for no behavior change).
  • Deferred: the multi-process tests (single node / multi node / owner on the same or another node, bitwise against reduce_scatter). They need GPUs and come with the acceptance run.

Not verified: none of these changes has run on a GPU. Every change is outside the numerical path (no kernel body, summation order, __fmul_rn / __fadd_rn order, release condition, credit target or coefficient placement changed); the 4-node bitwise acceptance run is still to come. Comments, docstrings, errors and logs of the series are now in English (one commit per layer; outside string literals the token stream / AST is identical, checked mechanically).

Changes after the second review

  • (Superseded below: record_window_mark now returns an error code.) record_window_mark's declaration in gin_engine.h lists its return values: the window mark slot (>= 0), or -1 when configure has not run, the compute stream cannot be read, or cudaEventRecord failed (same style as bucket_ready / tail_mark). Comment only, no behavior change.
  • cpplint fixes in this layer's files (casts, int64_t / uint64_t, pointer out-parameters, include order); details in [WIP][Distributed] gin_overlap 1/3: dynload wrappers, device kernels and the build gate #79967.
  • record_window_mark(int* slot) returns an error code and writes the slot to *slot (was: the slot, or -1 for every failure): cudaErrorNotReady without configure, the error of paddle_compute_stream, or the cudaError_t of cudaEventRecord. The declaration says so.

Not verified: the precision acceptance run (4 nodes, bitwise) is still in progress, and the changes of this round have not been through it. The behavior changes of this round are the argument check in release::setup (#79970) and the error codes of record_window_mark / tail_mark (#79969 / #79970, Python still gets the slot); the cpplint fixes (#79967) change casts, integer type spellings and reference -> pointer parameters only.

是否引起精度变化

否

🤖 Generated with Claude Code

@Paddle-Review-Bot

Paddle-Review-Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Paddle-Bot Review Board (review完成)

序号 位置 优先级 规则来源 状态
1 机内拷贝的批量 CE 调用签名 P1 仓库规则:Paddle 基础评审规则 ✅
2 scale_range 越界检查回绕 P2 仓库规则:Paddle 基础评审规则 ✅

可维护性建议(总分 85.3 / 100,只给建议,不影响评审结论)

序号 位置 维度 状态
1 窗口 mark 的返回码约定 错误处理一致性 ✅
2 专家 put 的偏移参数顺序 API 表面一致性 ✅
3 gather_setup 的错误码区分 逻辑清晰度 ✅
4 返回码透传的重复样板 底层优雅度 ✅
5 阶段提交函数缺少用例 测试策略 💡
各维度得分(仅评本 PR 改动,点击展开)
65 测试策略
5
85 API 表面一致性
2
85 约定偏离
85 跨模块架构
85 抽象适配度
85 依赖健康度
85 高层优雅度
85 类型安全
85 AI 生成债务
85 未完成迁移
85 命名质量
86 错误处理一致性
1
86 中层优雅度
86 初始化耦合
86 设计一致性
88 底层优雅度
4
89 契约一致性
89 逻辑清晰度
3
90 鉴权一致性
90 包组织结构

Powered by Nyanpasu claude with Opus 5.5 默认推理级别, please check the suggestions carefully.

Comment on lines +445 to +452
CUDA_CHECK(cudaMemcpyBatchAsync(dst_pointers.data(),
src_pointers.data(),
sizes.data(),
dst_pointers.size(),
&copy_attrs,
&fail_index,
1,
g_stream_intra_node_copy));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1

这里和第 889 行都按 CUDA 13.0 的 8 参数签名调用 cudaMemcpyBatchAsync。CUDA 12.8/12.9 的签名有 9 个参数:(void **dsts, void **srcs, size_t *sizes, size_t count, cudaMemcpyAttributes *attrs, size_t *attrsIdxs, size_t numAttrs, size_t *failIdx, cudaStream_t stream),cuda_runtime.h 里的两个模板重载分别是 9 个和 7 个参数。8 个实参匹配不到任何一个声明,src_pointers.data()(const void**)也不能隐式转成 void**。门控 collective/CMakeLists.txt:48 放行的是 CUDA_VERSION >= 12.8,所以在 12.8/12.9 上打开门控时这个文件编译不过。

另外,CUDA 13 下第 6 个实参的位置是 attrsIdxs,这里传的是名为 fail_index 的变量。它的值是 0,结果碰巧正确,但变量名会误导读者。

建议二选一:

  • 按 CUDART_VERSION 分两种写法。12.x 补上 attrsIdxs,并把 srcs 用 const_cast 转成 void**;13.x 保持现在的调用,把变量改名为 attrs_idx。
  • 把门控提到 CUDA_VERSION >= 13.0,同时修改该处的注释。

Comment on lines +750 to +751
if (!comm_group.send ||
send_offset + (size_t)numel * 4 > comm_group.send_bytes) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2

numel 是 Python 传进来的 long long。这里的越界检查 send_offset + (size_t)numel * 4 > send_bytes 会因为无符号回绕而失效。例如 numel = -1、send_offset = 8 时,左边回绕成 4,检查通过。随后 launch_scale_range 收到 numel = SIZE_MAX,kernel 会一直写到 send window 之外。blocks_for_numel 里的 (numel + 255) 也会回绕,结果是只用 1 个 block 越界写,最终表现为 illegal address 或静默踩坏显存。gather_put_to_owner 第 873 行的 send_offset + numel * 4 > comm_group.send_bytes 是同样的写法,而且它没有检查 shard_numel < 0。

建议两处都改成不会溢出的形式,例如:
if (numel < 0 || send_offset > send_bytes || (size_t)numel > (send_bytes - send_offset) / 4) return (int)cudaErrorInvalidValue;

Comment on lines +313 to +321
int GradCommEngine::record_window_mark() {
if (g_window_marks.empty()) return -1;
cudaStream_t compute_stream = nullptr;
if (paddle_compute_stream(&compute_stream)) return -1;
const int i = g_window_mark_index;
g_window_mark_index = (g_window_mark_index + 1) % (int)g_window_marks.size();
if (cudaEventRecord(g_window_marks[i], compute_stream) != cudaSuccess)
return -1;
return i;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(错误处理一致性,不阻塞合入):GradCommEngine 在头文件里约定返回码为“0 成功,<10000 为 cudaError_t”,但 record_window_mark 成功时返回槽号,失败时一律返回 -1。三种失败(mark 环未配置、取计算流失败、cudaEventRecord 失败)因此无法区分,-1 交给 error_string 也翻译不出原因。建议改成 int record_window_mark(int* slot),槽号走出参,返回值只放真实错误码(未配置时返回 cudaErrorNotReady,取流失败时返回 paddle_compute_stream 的返回码)。

Comment on lines +580 to +585
const int rc = launch_put_to_expert_peers(comm_group.dev_comm,
comm_group.win_recv,
comm_group.win_send,
comm_group.win_host,
recv_offset,
send_offset,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(API 表面一致性,不阻塞合入):两个结构相同的 put 封装,偏移参数顺序相反。launch_put_to_node_peers 收 (send_offset, recv_offset, ...)(第 540-541 行),这里的 launch_put_to_expert_peers 收 (recv_offset, send_offset, ...)。两个参数都是 size_t,以后改调用时写反了编译器不会报错。建议统一成同一顺序,例如都用 (send_offset, recv_offset, signal_offset),并同步修改 kernels.h 里的声明。

Comment on lines +795 to +796
if (!g_buckets_ready || g_gather_ready || num_slots < 0)
return (int)cudaErrorNotReady;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(逻辑清晰度,不阻塞合入):这里把状态检查和参数校验合在一个条件里,num_slots < 0 也返回 cudaErrorNotReady,报出的是“尚未就绪”,看不出是参数错误。第 869 行 gather_put_to_owner 对非法 group 也返回 NotReady,而同文件其他接口(alloc_windows、scale_range 等)遇到非法 group 都返回 cudaErrorInvalidValue。建议把两处都拆开:状态不满足时返回 NotReady,参数非法时返回 InvalidValue。

Comment on lines +453 to +460
{
const int rc = launch_signal_intra_arrive(arrive_slots,
ranks_per_node,
rank_in_node,
seq,
g_stream_intra_node_copy);
if (rc) return rc;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(底层优雅度,不阻塞合入):本文件有十几处同样的 { const int rc = launch_xxx(...); if (rc) return rc; } 匿名块,例如第 453-460、485-496、623-634、636-647、814-821 行。建议在 gin_engine.h 的 CUDA_CHECK / NCCL_CHECK 旁边加一个 GIN_CHECK(x)(do { const int _rc = (x); if (_rc) return _rc; } while (0)),把这些块都换成一行,提交函数的主流程会清楚很多。

Comment on lines +404 to +410
int submit_dense_intra_node_copy(int bucket,
size_t recv_offset,
size_t send_offset,
long long shard_numel,
cudaEvent_t mark_event,
size_t intra_arrive_offset,
unsigned long long seq) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 可维护性建议(测试策略,不阻塞合入):这一层新增了六个阶段提交函数和 2D gather,有 offset/numel 越界检查和多条按拓扑分支的路径(ranks_per_node < 2、num_nodes == 1、owner 在同节点或跨节点),但目前没有任何用例覆盖。PR 描述说测试会在后续 PR 补上。建议后续的多进程用例至少覆盖单节点、多节点、owner 同节点和跨节点四种组合,和 reduce_scatter / all_gather 的结果逐位比对,再加一个非法 offset/numel 的入参用例。

Paddle links neither libnccl nor libcuda, so new code that calls them has to go
through the phi::dynload dlopen wrappers. The gin_overlap gradient
communication added in the following commits needs three symbols that are not
wrapped yet:

* ncclDevCommCreate / ncclGetLsaDevicePointer, the two host entry points of
  NCCL's device (LSA/GIN) API. They are declared in nccl_device.h, which also
  carries the device-side inline implementations of the whole device API, so
  including it from nccl.h would land all of that in every translation unit
  that reaches for any NCCL symbol. The declarations it needs are repeated
  here instead, the way the internal entry points at the top of the file
  already are, and are skipped in translation units that include
  nccl_device.h themselves. Same 2.30.7 floor as the zero-SM group above
  them: below that the feature is refused, not degraded.
* cuMemcpy2DAsync, for the pitched copy that splits a fused all_gather back
  into per-bucket storages. It is wrapped under the name libcuda exports,
  cuMemcpy2DAsync_v2: the wrapper dlsym()s the name it is given but types
  itself from decltype(&::name), which cuda.h's macro resolves to the v2
  declaration, so passing the plain name would call the v1 entry point (whose
  parameter is the old CUDA_MEMCPY2D_v1 layout) through a v2 signature. Call
  sites keep writing cuMemcpy2DAsync and let cuda.h's macro rewrite it.

The host overload of ncclTeamWorld is deliberately not wrapped: it shares a
name with the device-side ncclTeamWorld(ncclDevComm const&), so
decltype(&::ncclTeamWorld) cannot be formed, and no host code needs it.

Co-Authored-By: Claude Code <noreply@anthropic.com>
ncclDevCommCreate and ncclGetLsaDevicePointer need the same NCCL 2.30.7
as the other symbols of NCCL_RAND_ROUTINE_EACH_AFTER_23007, so they join
that group instead of a separate NCCL_RAND_ROUTINE_EACH_DEVICE_API under
an identical #if. cuMemcpy2DAsync_v2 has no version gate and joins
CUDA_ROUTINE_EACH; CUDA_ROUTINE_EACH_MEMCPY and its two expansions go.

The comment on the nccl_device.h prototypes now states what actually
holds: the prototypes must match nccl_device.h exactly, because this
header is usually included first and both declarations then coexist;
the guard only drops the duplicate in the other include order.

Co-Authored-By: Claude Code <noreply@anthropic.com>
….30.7 gate

Co-Authored-By: Claude Code <noreply@anthropic.com>
@AlAuAu
AlAuAu force-pushed the gin_split_4_engine_submit branch from d44987d to c61dbf0 Compare November 21, 2026 11:21
@AlAuAu
AlAuAu force-pushed the gin_split_4_engine_submit branch from c61dbf0 to fd3f49c Compare November 21, 2026 12:43
First piece of moving the NCCL-GIN sharding-stage1 gradient primitives out of
PaddleFleet's extension (paddlefleet_ops/gin_overlap/csrc) into Paddle: the
CUDA kernels and their launch wrappers, plus the CMake gate that decides
whether the directory is built at all. Nothing calls these yet; the host-side
engine that launches them comes in the next commits.

* kernels/reduce.cu: the intra-node column sum, the dense shard sum across
  nodes and the expert shard sum, each scaled in place.
* kernels/put.cu: GIN one-sided puts to the same-rank peers on other nodes,
  to the expert-group peers, to the 2D-bucket owner, and the GIN flush.
* kernels/signal.cu: the intra-node counter / arrival writes over LSA
  pointers, the GIN credit signal, a single release store, and the lookup of
  the host-window LSA pointers.
* kernels/scale.cu: an in-place scale of a send-window range.
* kernels/kernels.h: the launch_* / preload_* declarations the host side is
  allowed to call (no <<<>>> outside kernels/).
* gin_dynload.h: brings the phi::dynload wrappers (previous commit) into
  namespace gin_overlap under their plain names, so the host code that comes
  next calls NCCL / the driver through dlopen without touching call sites.

The code is moved, not rewritten: the sum order, the kernels and the block
counts are untouched; the code-level diff against the extension is include
paths, namespaces and include guards (#pragma once). Two casts of CUDA
built-ins are respelled as static_cast so that the
no-int64-type-cuda-index-without-cast ast-grep rule sees them; cpplint is
switched off for the directory, as it is for deep_ep. The comments say what
each kernel does and what must hold; reduce.cu's header lists what keeps the
result bit-identical to reduce_scatter (ring order starting at own rank + 1,
__fmul_rn before __fadd_rn) and states that the block / thread count is not
among them: it only decides which indices a thread walks. The caller does
change the reduce kernels' block count between windows within one run, and
the bit-exactness check against reduce_scatter passes with that.

Build gate (CMake is written but not run in this change):
* arch: only built when NVCC_ARCH_BIN contains 9.0 or newer, computed the
  way paddle/phi/kernels/CMakeLists.txt computes has_arch_ge80. Inside the
  directory the <= 8.x gencodes are stripped off CMAKE_CUDA_FLAGS, as
  deep_ep/CMakeLists.txt does.
* NCCL >= 2.30.7 with nccl_device.h present, and CUDA >= 12.8 for
  cudaMemcpyBatchAsync.
The result is cached as WITH_GIN_OVERLAP for the pybind layer added later.
gin_dynload.h #errors below the NCCL floor as a backstop for the gate.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Translate every comment, docstring, error message and log line of the
gin_overlap sources (C++, CUDA, CMake, pybind and the Python layer) into
English, and tighten the wording on the way. The terms are defined once,
in the gin_engine.h file header (C++) and in the engine.py module
docstring (Python); every other file uses them as defined there.

Strings only. comment_clean_check.py --ignore-strings passes on every
changed file: C++ / CUDA / CMake token streams are identical except for
string literals (printf conversions inside merged literals keep their
order), and the Python ASTs are identical except for str constants and
the literal parts of f-strings. Log lines all carry the [gin_overlap]
prefix; copy_2d errors now start with "copy_2d:".

Co-Authored-By: Claude Code <noreply@anthropic.com>
…the gate

Both cudaMemcpyBatchAsync calls use the CUDA 13 signature (8 arguments,
const void** sources). CUDA 12.8 / 12.9 declare a 9-argument version, so
a build with the gate open on those toolkits would not compile. Raise the
gate to CUDA >= 13.0, the only toolkit the code has been built and run
with, instead of adding an untested 12.x branch; gin_dynload.h gets the
matching #error backstop. The batch attribute index is named attrs_idx,
which is what that argument is.

The arch condition reuses ARCH_BIN_CONTAINS_GE_90, computed at the top of
the same file for DeepEP, instead of a second loop over NVCC_ARCH_BIN.
_BUILD_HINT in the Python layer now lists exactly the gate's conditions,
and the CMake comment says to change both together.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…ne API

Argument checks (values come from Python):
- scale_range, gather_put_to_owner, send_window_tensor and copy_2d check
  their bounds without unsigned wrap-around, and reject numel < 0.
  copy_2d also rejects non-contiguous tensors (a stride-0 or transposed
  view passes a numel-based bound but the copy reads contiguous bytes).
- release::setup checks every offset of the per-bucket tables and the
  host window areas: non-negative, aligned (counters 8 B, data 4 B) and
  inside its window. setup_gather_table checks both arrival counter
  offsets of each entry the same way. CommGroup records recv_bytes for
  this, and alloc_symmetric_window returns the size it actually
  registered instead of the caller repeating the 4096 B minimum.
- A wrong state returns cudaErrorNotReady and a bad argument
  cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
- bucket_ready rejects a bucket reported twice in a step in the RUN state
  too, from a bitmap only the main thread touches.

Init failures:
- configure destroys the streams and events it created, frees its
  temporary buffer and resets the globals it set when a later step fails,
  so it can be retried. set_comms clears the groups it set on failure.
  release::setup unpublishes its control block if the tail thread cannot
  be started. The steps that cannot be rolled back (window registration,
  devcomm creation, the host window) say so: restart the process.
- The gin_engine.h constraints state that the engine lives as long as
  the process; shutdown_process_group / restart_process_group are not
  supported.

API:
- GIN_CHECK replaces the dozen { rc = launch_x(...); if (rc) return rc; }
  blocks. launch_* / preload_* return cudaError_t.
- launch_put_to_expert_peers takes (send_offset, recv_offset) like
  launch_put_to_node_peers.
- join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert),
  as bucket_ready and the gather table do, instead of is_dense.
- The six release positions are named variables, one per stage.
- The methods that do not return a code (record_window_mark, tail_mark,
  bucket_ready, tail_all_posted, tail_wait, tail_abort, release_activity,
  the version getters) document what they return.
- tail_wait / tail_abort release the GIL while they block.

No kernel, reduction order, release condition or credit target changes.

Co-Authored-By: Claude Code <noreply@anthropic.com>
kernels/release.cu has no kernel and no <<<>>>: it is the host-side
release state machine and its std::thread. kernels/ holds only the
kernel launch wrappers, so the file moves next to gin_engine.cc as
release.cc and is compiled by the host compiler instead of nvcc. The
content is unchanged apart from the file name in comments.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Remove gin_overlap/ from the cpplint exclude list and fix every finding:
C-style casts become static_cast / reinterpret_cast, long long and
unsigned long long become int64_t / uint64_t, non-const reference
out-parameters become pointers (call sites updated), and C system headers
come before C++ ones. atomicAdd_system keeps its unsigned long long
prototype through one aliased type. No change to values, types seen by
Python, or arithmetic.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…+ sources only

The four warning options are wrapped in $<$<COMPILE_LANGUAGE:CXX>:...>, so
nvcc no longer receives them for kernels/*.cu.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…treams

Second piece of the move from PaddleFleet's extension: the host side that
builds everything the per-step code runs on. Nothing calls it yet (the
Python binding comes last), and nothing is submitted to the GPU per step
yet (next commit).

* gin_engine.h / gin_engine.cc, initialization only: the CommGroup state of
  the two communication groups (sharding / moe_sharding), the error-check
  macros, and the GradCommEngine methods that the Python layer calls once at
  startup, in this order: set_comms (reuse Paddle's ncclComm_t for each
  ring id and cross-check size / rank), alloc_windows (send / recv / signal
  as NCCL_WIN_COLL_SYMMETRIC windows), alloc_host_counters (one HOST_NUMA
  host window per group, registered with NCCL_WIN_STRICT_ORDERING),
  create_dev_comm (ncclDevCommCreate + LSA pointers), set_group_topology,
  and configure (side streams, per-bucket events, the window-mark ring, the
  credit offsets, the host-window LSA pointers, and preloading every kernel
  so that the first launch of a module does not land on the backward
  critical path). Also error_string and the NCCL runtime / build version
  queries.
* paddle_comm.cc: takes the ncclComm_t out of Paddle's ProcessGroup by ring
  id, the way DeepEP does (ProcessGroupMapFromGid -> ProcessGroupNCCL ->
  NCCLCommContext::GetNcclComm).
* paddle_tensor.cc / .h: Paddle's current compute stream, the zero-copy
  paddle::Tensor over a send-window slice (paddle::from_blob with deleter =
  nullptr, so the memory stays owned by the NCCL window, plus a post-wrap
  self-check), and the pitched copy_2d through cuMemcpy2DAsync.

The file header of gin_engine.h is the one full description of the C++
side (terms, the two groups, the dense / expert / 2D stages, the receive
pools, the host-window layout and the constraints). It is added here in
full: it is organised by concept, not by layer, so the parts about the
per-stage submits and the release layer (next two commits) cannot be
moved out without rewriting it.

paddle_comm.cu and paddle_tensor.cu become .cc: they were .cu only because
an out-of-tree build has to hand any translation unit that includes Paddle
headers to nvcc (thrust); in-tree they are ordinary sources. The pybind
registration that was stranded in paddle_tensor.cu moves to the pybind
layer, so paddle_tensor.h declares the real paddle::Tensor signatures.

The library now also depends on process_group_nccl, for paddle_comm.cc.
Like the kernels, the code is moved, not rewritten: same checks, same
order of collective calls.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Translate every comment, docstring, error message and log line of the
gin_overlap sources (C++, CUDA, CMake, pybind and the Python layer) into
English, and tighten the wording on the way. The terms are defined once,
in the gin_engine.h file header (C++) and in the engine.py module
docstring (Python); every other file uses them as defined there.

Strings only. comment_clean_check.py --ignore-strings passes on every
changed file: C++ / CUDA / CMake token streams are identical except for
string literals (printf conversions inside merged literals keep their
order), and the Python ASTs are identical except for str constants and
the literal parts of f-strings. Log lines all carry the [gin_overlap]
prefix; copy_2d errors now start with "copy_2d:".

Co-Authored-By: Claude Code <noreply@anthropic.com>
…ne API

Argument checks (values come from Python):
- scale_range, gather_put_to_owner, send_window_tensor and copy_2d check
  their bounds without unsigned wrap-around, and reject numel < 0.
  copy_2d also rejects non-contiguous tensors (a stride-0 or transposed
  view passes a numel-based bound but the copy reads contiguous bytes).
- release::setup checks every offset of the per-bucket tables and the
  host window areas: non-negative, aligned (counters 8 B, data 4 B) and
  inside its window. setup_gather_table checks both arrival counter
  offsets of each entry the same way. CommGroup records recv_bytes for
  this, and alloc_symmetric_window returns the size it actually
  registered instead of the caller repeating the 4096 B minimum.
- A wrong state returns cudaErrorNotReady and a bad argument
  cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
- bucket_ready rejects a bucket reported twice in a step in the RUN state
  too, from a bitmap only the main thread touches.

Init failures:
- configure destroys the streams and events it created, frees its
  temporary buffer and resets the globals it set when a later step fails,
  so it can be retried. set_comms clears the groups it set on failure.
  release::setup unpublishes its control block if the tail thread cannot
  be started. The steps that cannot be rolled back (window registration,
  devcomm creation, the host window) say so: restart the process.
- The gin_engine.h constraints state that the engine lives as long as
  the process; shutdown_process_group / restart_process_group are not
  supported.

API:
- GIN_CHECK replaces the dozen { rc = launch_x(...); if (rc) return rc; }
  blocks. launch_* / preload_* return cudaError_t.
- launch_put_to_expert_peers takes (send_offset, recv_offset) like
  launch_put_to_node_peers.
- join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert),
  as bucket_ready and the gather table do, instead of is_dense.
- The six release positions are named variables, one per stage.
- The methods that do not return a code (record_window_mark, tail_mark,
  bucket_ready, tail_all_posted, tail_wait, tail_abort, release_activity,
  the version getters) document what they return.
- tail_wait / tail_abort release the GIL while they block.

No kernel, reduction order, release condition or credit target changes.

Co-Authored-By: Claude Code <noreply@anthropic.com>
kernels/release.cu has no kernel and no <<<>>>: it is the host-side
release state machine and its std::thread. kernels/ holds only the
kernel launch wrappers, so the file moves next to gin_engine.cc as
release.cc and is compiled by the host compiler instead of nvcc. The
content is unchanged apart from the file name in comments.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Remove gin_overlap/ from the cpplint exclude list and fix every finding:
C-style casts become static_cast / reinterpret_cast, long long and
unsigned long long become int64_t / uint64_t, non-const reference
out-parameters become pointers (call sites updated), and C system headers
come before C++ ones. atomicAdd_system keeps its unsigned long long
prototype through one aliased type. No change to values, types seen by
Python, or arithmetic.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…e slot as an out-parameter

C++: both return 0 or a cudaError_t code and write the slot to *slot:
cudaErrorNotReady without configure / release_setup, the error of
paddle_compute_stream or cudaEventRecord, and cudaErrorLaunchOutOfResources
when this step's tail mark slots are used up (was -1 / -2). The Python
binding still returns the slot and raises RuntimeError with the error text
on failure. CPU test of the binding before setup.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…ther

Third piece of the move from PaddleFleet's extension: the host functions
that hand one stage of one bucket to its side stream, and the stream /
event plumbing around them. Still no caller: the release layer that decides
when each stage may go (next commit) is the only code that calls submit_*.

* The six stage submits: submit_dense_intra_node_copy (cudaMemcpyBatchAsync
  of the column to the other ranks on the node + the arrival write),
  submit_dense_intra_node_reduce (+ intra-node credit),
  submit_dense_inter_node_put, submit_moe_put, submit_dense_inter_node_reduce
  and submit_moe_reduce (+ GIN credit). Each one only waits on work this
  rank has already handed out (a compute-stream mark or a local side-stream
  event), never on a remote counter: whether the remote side is ready is
  decided on the host from the host window before the call.
* record_window_mark: records a mark on Paddle's current compute stream into
  the ring configure() created.
* join_bucket_streams (compute stream waits for a bucket's side-stream
  events), scale_range, end_step (GIN flush of both groups so the send
  window can be reused), stats.
* Muon 2D-bucket gather to the owner: gather_setup, gather_gate,
  gather_wait_bucket, gather_mark_ready, gather_put_to_owner (GIN put across
  nodes, CE + release store within a node) and gather_join. The table-driven
  entry point, gather_put_entry, forwards to the release layer and comes
  with it.

Moved as is from the extension, like the previous commits.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Translate every comment, docstring, error message and log line of the
gin_overlap sources (C++, CUDA, CMake, pybind and the Python layer) into
English, and tighten the wording on the way. The terms are defined once,
in the gin_engine.h file header (C++) and in the engine.py module
docstring (Python); every other file uses them as defined there.

Strings only. comment_clean_check.py --ignore-strings passes on every
changed file: C++ / CUDA / CMake token streams are identical except for
string literals (printf conversions inside merged literals keep their
order), and the Python ASTs are identical except for str constants and
the literal parts of f-strings. Log lines all carry the [gin_overlap]
prefix; copy_2d errors now start with "copy_2d:".

Co-Authored-By: Claude Code <noreply@anthropic.com>
…the gate

Both cudaMemcpyBatchAsync calls use the CUDA 13 signature (8 arguments,
const void** sources). CUDA 12.8 / 12.9 declare a 9-argument version, so
a build with the gate open on those toolkits would not compile. Raise the
gate to CUDA >= 13.0, the only toolkit the code has been built and run
with, instead of adding an untested 12.x branch; gin_dynload.h gets the
matching #error backstop. The batch attribute index is named attrs_idx,
which is what that argument is.

The arch condition reuses ARCH_BIN_CONTAINS_GE_90, computed at the top of
the same file for DeepEP, instead of a second loop over NVCC_ARCH_BIN.
_BUILD_HINT in the Python layer now lists exactly the gate's conditions,
and the CMake comment says to change both together.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…ne API

Argument checks (values come from Python):
- scale_range, gather_put_to_owner, send_window_tensor and copy_2d check
  their bounds without unsigned wrap-around, and reject numel < 0.
  copy_2d also rejects non-contiguous tensors (a stride-0 or transposed
  view passes a numel-based bound but the copy reads contiguous bytes).
- release::setup checks every offset of the per-bucket tables and the
  host window areas: non-negative, aligned (counters 8 B, data 4 B) and
  inside its window. setup_gather_table checks both arrival counter
  offsets of each entry the same way. CommGroup records recv_bytes for
  this, and alloc_symmetric_window returns the size it actually
  registered instead of the caller repeating the 4096 B minimum.
- A wrong state returns cudaErrorNotReady and a bad argument
  cudaErrorInvalidValue (gather_setup, gather_put_to_owner, scale_range).
- bucket_ready rejects a bucket reported twice in a step in the RUN state
  too, from a bitmap only the main thread touches.

Init failures:
- configure destroys the streams and events it created, frees its
  temporary buffer and resets the globals it set when a later step fails,
  so it can be retried. set_comms clears the groups it set on failure.
  release::setup unpublishes its control block if the tail thread cannot
  be started. The steps that cannot be rolled back (window registration,
  devcomm creation, the host window) say so: restart the process.
- The gin_engine.h constraints state that the engine lives as long as
  the process; shutdown_process_group / restart_process_group are not
  supported.

API:
- GIN_CHECK replaces the dozen { rc = launch_x(...); if (rc) return rc; }
  blocks. launch_* / preload_* return cudaError_t.
- launch_put_to_expert_peers takes (send_offset, recv_offset) like
  launch_put_to_node_peers.
- join_bucket_streams / gather_wait_bucket take kind (0 dense, 1 expert),
  as bucket_ready and the gather table do, instead of is_dense.
- The six release positions are named variables, one per stage.
- The methods that do not return a code (record_window_mark, tail_mark,
  bucket_ready, tail_all_posted, tail_wait, tail_abort, release_activity,
  the version getters) document what they return.
- tail_wait / tail_abort release the GIL while they block.

No kernel, reduction order, release condition or credit target changes.

Co-Authored-By: Claude Code <noreply@anthropic.com>
kernels/release.cu has no kernel and no <<<>>>: it is the host-side
release state machine and its std::thread. kernels/ holds only the
kernel launch wrappers, so the file moves next to gin_engine.cc as
release.cc and is compiled by the host compiler instead of nvcc. The
content is unchanged apart from the file name in comments.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Remove gin_overlap/ from the cpplint exclude list and fix every finding:
C-style casts become static_cast / reinterpret_cast, long long and
unsigned long long become int64_t / uint64_t, non-const reference
out-parameters become pointers (call sites updated), and C system headers
come before C++ ones. atomicAdd_system keeps its unsigned long long
prototype through one aliased type. No change to values, types seen by
Python, or arithmetic.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…e slot as an out-parameter

C++: both return 0 or a cudaError_t code and write the slot to *slot:
cudaErrorNotReady without configure / release_setup, the error of
paddle_compute_stream or cudaEventRecord, and cudaErrorLaunchOutOfResources
when this step's tail mark slots are used up (was -1 / -2). The Python
binding still returns the slot and raises RuntimeError with the error text
on failure. CPU test of the binding before setup.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@AlAuAu
AlAuAu force-pushed the gin_split_4_engine_submit branch from fd3f49c to 3406e43 Compare November 21, 2026 14:52
@AlAuAu AlAuAu closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants