Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
33ac8e3
[None][feat] Wide-EP fault tolerance: enable_wide_ep_fault_tolerance …
arusli-nv Nov 18, 2026
72315d5
[None][test] Set enable_wide_ep_fault_tolerance in the model loader t…
arusli-nv Nov 19, 2026
93aceb4
[None][fix] Address the review of the fault-tolerant MoE all-to-all
arusli-nv Nov 19, 2026
20cc6c3
[None][test] Set enable_wide_ep_fault_tolerance in the DSpark EPLB te…
arusli-nv Nov 19, 2026
58b4e7e
[None][feat] EPLB: unbacked spare expert slots and expert restore int…
arusli-nv Nov 19, 2026
1650f13
[None][feat] Wide-EP fault tolerance: membership record, authority pr…
arusli-nv Nov 19, 2026
1c08963
[None][fix] Mark the fault-tolerance shared-secret variable name for …
arusli-nv Nov 20, 2026
be180d2
[None][fix] Reload the slots that a stopped expert restore wrote
arusli-nv Nov 20, 2026
df80eca
Merge PR 2 fix be180d2b3a: reload the slots that a stopped expert res…
arusli-nv Nov 20, 2026
d347094
[None][fix] Release only the spare slots that the installed placement…
arusli-nv Nov 21, 2026
fe7eab1
Merge PR 2 fix d3470947d6: release only the spare slots that the plac…
arusli-nv Nov 21, 2026
f08dcfe
[None][fix] Count a joining rank's READY when finding ranks that did …
arusli-nv Nov 21, 2026
fa40007
[None][fix] Wait on liveness sockets with selectors
arusli-nv Nov 21, 2026
2efb442
[None][docs] State what the resumed epoch of a replaced INSTALL means…
arusli-nv Nov 21, 2026
7a50994
[None][docs] State the spare-slot contract of a MoE backend
arusli-nv Nov 21, 2026
4df2ae4
Merge PR 2 head 7a50994a09: the spare-slot contract in the MoE develo…
arusli-nv Nov 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
505 changes: 442 additions & 63 deletions cpp/tensorrt_llm/kernels/moe/communication/moeAlltoAllKernels.cu

Large diffs are not rendered by default.

102 changes: 84 additions & 18 deletions cpp/tensorrt_llm/kernels/moe/communication/moeAlltoAllKernels.h
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
#pragma once
#include "tensorrt_llm/common/config.h"
#include "tensorrt_llm/common/tllmDataType.h"
#include <cstddef>
#include <cuda_bf16.h>
#include <cuda_fp16.h>

Expand All @@ -37,6 +38,42 @@ static_assert(kRankMaskWords * 64 >= kMaxRanks, "active_rank_mask too small for
static constexpr int64_t kAssumedClockHz = 2000ll * 1000ll * 1000ll;
static constexpr int64_t kDefaultTimeoutCycles = 300ll * kAssumedClockHz;

// Waits on a peer shorter than kAbortPollCycles never read the host abort state.
// Longer waits read it once per kAbortPollCycles per block.
static constexpr int64_t kAbortPollCycles = 100LL * 1000LL;

// A wait on a peer longer than kPublishWaitedFlagCycles publishes the flag value it waits for in
// MoeA2AAbortState::waited_flag, so the host can tell which peer this rank waits on.
static constexpr int64_t kPublishWaitedFlagCycles = 20LL * 1000LL * 1000LL;

//! \brief Abort state of the rank-mask collectives, in pinned host memory.
//!
//! The host sets `requested` to end the waits on peers. A kernel that sees it stops waiting and sets `status` to
//! kCancelled. While `requested` or `status` is nonzero, the outputs of every collective that uses the abort state
//! are invalid. The incomplete-round word below decides what the following collectives exchange.
struct MoeA2AAbortState
{
static constexpr int32_t kCancelled = 1;

int32_t requested;
//! kCancelled once a kernel stopped waiting because of `requested`, or the incomplete-round check found the word
//! set, 0 otherwise
int32_t status;
//! Per peer, the round flag value that a long wait on the peer waits for, 0 otherwise
uint32_t waited_flag[kMaxRanks];
};

// Incomplete-round word: a uint32_t in the workspace of each rank of a layout built with a rank mask. A collective
// that ends without the data of some active source stores its incomplete-round value (its round flag value with bit 0
// set, so never 0) there, unless the word is already nonzero. The word keeps that value until the workspace is reset.
// While it is nonzero, every later collective of this rank exchanges with this rank only, so no peer reads rows of an
// incomplete round. The MoE after a dispatch may still overwrite rows of this rank that an active peer's fence combine
// reads. So a fence dispatch that aborts, or that starts with the word set, first stores the value in the word of
// every active rank, and a check after each forward that finds its word set marks the abort state cancelled
// (moe_a2a_check_incomplete_round_launch). The incomplete-round value of round N's combine, which round N + 1's
// dispatch shares, is ignored within round N: this rank's combine or a peer's next dispatch may store it during the
// round, and the prepare, the sanitize and every block of the combine must exchange with the same peers.

// Describes a single payload type to be communicated
struct PayloadDescriptor
{
Expand Down Expand Up @@ -76,8 +113,15 @@ struct DispatchKernelPointers
// Active-rank bitmask: bit i set => rank i is alive and participates in this collective.
// Word w covers ranks 64*w..64*w+63 for w in [0, kRankMaskWords). Tokens routed to a masked
// rank are dropped (topk_*[k] = -1); flag writes/waits to/from masked peers are skipped.
// The local rank's own bit must always be set; this is checked at launch time.
uint64_t active_rank_mask[kRankMaskWords];
// The address is fixed, so captured CUDA graphs read a mask rewritten in place while no collective runs.
uint64_t const* active_rank_mask{nullptr};
// Byte offset from completion_flags[rank] to the incomplete-round word (see above) of that rank, or 0 without one.
// Every rank's workspace has the same layout, so one offset locates the word of every rank.
int64_t incomplete_round_offset{0};
MoeA2AAbortState* abort{nullptr};
// The rank-mask fields above fill kRankMaskWords words, so the parameter offsets of the fields and of the kernel
// arguments after them do not depend on them.
uint64_t rank_mask_reserved{0};

// Host-selected wait budget in clock64() cycles.
int64_t timeout_cycles{kDefaultTimeoutCycles};
Expand All @@ -101,12 +145,28 @@ struct CombineKernelPointers

// Active-rank bitmask: see DispatchKernelPointers::active_rank_mask. Combine skips
// completion flag writes/waits to/from inactive peers.
uint64_t active_rank_mask[kRankMaskWords];
uint64_t const* active_rank_mask{nullptr};
// Incomplete-round word in the local workspace (see above), or nullptr without one.
uint32_t* incomplete_round{nullptr};
MoeA2AAbortState* abort{nullptr};
// The rank-mask fields above fill kRankMaskWords words, as in DispatchKernelPointers.
uint64_t rank_mask_reserved{0};

// Host-selected wait budget in clock64() cycles.
int64_t timeout_cycles{kDefaultTimeoutCycles};
};

// The kernels take these structs by value, followed by scalar arguments, so the rank-mask fields keep a fixed
// kRankMaskWords-word slot and timeout_cycles stays the last field. The parameter offsets of timeout_cycles and of
// the scalar arguments do not depend on the rank-mask fields, and neither does the generated code of the kernels
// without a rank mask.
static_assert(offsetof(DispatchKernelPointers, timeout_cycles)
== offsetof(DispatchKernelPointers, active_rank_mask) + kRankMaskWords * sizeof(uint64_t));
static_assert(sizeof(DispatchKernelPointers) == offsetof(DispatchKernelPointers, timeout_cycles) + sizeof(int64_t));
static_assert(offsetof(CombineKernelPointers, timeout_cycles)
== offsetof(CombineKernelPointers, active_rank_mask) + kRankMaskWords * sizeof(uint64_t));
static_assert(sizeof(CombineKernelPointers) == offsetof(CombineKernelPointers, timeout_cycles) + sizeof(int64_t));

// Dispatch phase parameters
struct MoeA2ADispatchParams
{
Expand Down Expand Up @@ -142,6 +202,8 @@ struct MoeA2ADispatchParams
int* recv_counters[kMaxRanks];
uint32_t* completion_flags[kMaxRanks]; // If completion_flags[target_rank][source_rank] == *flag_val, then source
// rank has signaled the target rank
// Byte offset from completion_flags[rank] to the incomplete-round word (see above) of that rank, or 0 without one.
int64_t incomplete_round_offset{0};
void* recv_buffers[kMaxRanks][kMaxPayloads]; // Per-rank receive buffers for each payload

// Optional: Statistics for EPLB
Expand All @@ -150,15 +212,12 @@ struct MoeA2ADispatchParams
int const* eplb_local_stats; // [eplb_stats_num_experts]
int* eplb_gathered_stats[kMaxRanks]; // [ep_size, eplb_stats_num_experts] per rank

// Whether to instantiate a kernel with active-rank checks.
// This is a launch-lifetime mode, independent of future execution-abort handling.
// Whether to instantiate a kernel with active-rank checks. An abort state requires it.
bool enable_rank_mask{false};

// Active-rank bitmask: see DispatchKernelPointers::active_rank_mask. Used only when
// enable_rank_mask is true; defaults to all-ones for backwards-compatible behavior.
// The mask is copied by value into kernel arguments. Rank-mask mode must reject
// CUDA graph replay until generation-scoped invalidation and recapture are available.
uint64_t active_rank_mask[kRankMaskWords] = {~uint64_t{0}, ~uint64_t{0}, ~uint64_t{0}, ~uint64_t{0}};
// Active-rank bitmask: see DispatchKernelPointers::active_rank_mask. Required when enable_rank_mask is true.
uint64_t const* active_rank_mask{nullptr};
MoeA2AAbortState* abort{nullptr};

// Host-selected wait budget in clock64() cycles.
int64_t timeout_cycles{kDefaultTimeoutCycles};
Expand Down Expand Up @@ -205,6 +264,8 @@ struct MoeA2ACombineParams
// per k, -1 for duplicates
int* topk_target_indices; // Top-K compact routing info per local token (size: [local_num_tokens, top_k]), dst index
// per k, -1 for duplicates
// Incomplete-round word in the local workspace (see above), or nullptr without one.
uint32_t* incomplete_round{nullptr};
// Local recv_counters[parity][source_rank]. The two parity banks alternate
// between A2A rounds.
int const* recv_counters;
Expand All @@ -214,15 +275,12 @@ struct MoeA2ACombineParams
// rank has signaled the target rank
uint8_t* combine_input_buffers[kMaxRanks]; // Expert-output/staging region on each rank

// Whether to instantiate a kernel with active-rank checks in peer synchronization.
// This is a launch-lifetime mode, independent of future execution-abort handling.
// Whether to instantiate a kernel with active-rank checks in peer synchronization. An abort state requires it.
bool enable_rank_mask{false};

// Active-rank bitmask: see DispatchKernelPointers::active_rank_mask. Used only when
// enable_rank_mask is true; defaults to all-ones for backwards-compatible behavior.
// The mask is copied by value into kernel arguments. Rank-mask mode must reject
// CUDA graph replay until generation-scoped invalidation and recapture are available.
uint64_t active_rank_mask[kRankMaskWords] = {~uint64_t{0}, ~uint64_t{0}, ~uint64_t{0}, ~uint64_t{0}};
// Active-rank bitmask: see DispatchKernelPointers::active_rank_mask. Required when enable_rank_mask is true.
uint64_t const* active_rank_mask{nullptr};
MoeA2AAbortState* abort{nullptr};

// Host-selected wait budget in clock64() cycles.
int64_t timeout_cycles{kDefaultTimeoutCycles};
Expand All @@ -236,12 +294,20 @@ void moe_a2a_combine_launch(MoeA2ACombineParams const& params);

void moe_a2a_prepare_combine_launch(MoeA2ACombineParams const& params);

//! Mark `abort` cancelled if the incomplete-round word (see above) is set, ordered after every earlier fence combine
//! in `stream`. Launch once per forward, after its last combine.
void moe_a2a_check_incomplete_round_launch(
uint32_t const* incomplete_round, MoeA2AAbortState* abort, cudaStream_t stream);

// Sanitize expert IDs for invalid tokens
// expert_ids: [ep_size, max_tokens_per_rank, top_k] (int32)
// recv_counters: [2, ep_size] (int32), number of valid tokens per source
// incomplete_round: incomplete-round word (see above), or nullptr. While it holds a nonzero value other than the
// incomplete-round value of this round's combine, only the tokens from ep_rank are valid.
// invalid_id: value to fill for invalid tokens' expert ids
void moe_a2a_sanitize_expert_ids_launch(int32_t* expert_ids, int32_t const* recv_counters, uint32_t const* flag_val,
int32_t invalid_id, int ep_size, int max_tokens_per_rank, int top_k, cudaStream_t stream);
uint32_t const* incomplete_round, int32_t invalid_id, int ep_rank, int ep_size, int max_tokens_per_rank, int top_k,
cudaStream_t stream);

} // namespace kernels::moe_comm

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -624,7 +624,8 @@ __global__ void moeComputeRouteSortKernel(MoeLoadBalanceMetaInfo metaInfo, MoePl
}

void moeComputeRouteDevice(MoeLoadBalanceMetaInfo metaInfo, MoePlacementInfo placementInfo,
int* const tokenSelectedExperts, int* tokenRoutedSlotIds, int tokenCount, bool offsetByEpRank, cudaStream_t stream)
int* const tokenSelectedExperts, int* tokenRoutedSlotIds, int tokenCount, bool offsetByEpRank, cudaStream_t stream,
bool singleReplicaPerExpert)
{
constexpr int kThreadCount = 256;
constexpr int kEltPerThread = 4;
Expand All @@ -635,7 +636,9 @@ void moeComputeRouteDevice(MoeLoadBalanceMetaInfo metaInfo, MoePlacementInfo pla
return;
}
int dynamicShmSize = sizeof(int16_t) * metaInfo.epSize * metaInfo.slotCountPerRank;
if (metaInfo.expertCount == metaInfo.epSize * metaInfo.slotCountPerRank)
// With a single replica per expert, expertReplicaStartOffset[e] is e, so globalSlotIds[e] is expert e's only slot.
// The direct kernel is then correct even when empty spare slots make the slots outnumber the experts.
if (singleReplicaPerExpert || metaInfo.expertCount == metaInfo.epSize * metaInfo.slotCountPerRank)
{
auto* kernelFn = moeComputeRouteNoRedundantKernel<1024, kThreadCount, kEltPerThread>;
// no redundant expert, so we don't need complex routing, but just assign to the correct solt.
Expand All @@ -650,15 +653,20 @@ void moeComputeRouteDevice(MoeLoadBalanceMetaInfo metaInfo, MoePlacementInfo pla
}
}

void moeWaitSignalForCpuStageHost(MoeLoadBalanceSingleLayerSignal* signal)
bool moeWaitSignalForCpuStageHost(MoeLoadBalanceSingleLayerSignal* signal, std::atomic<bool> const* stop)
{
bool ready = false;
do
{
auto loaded = __atomic_load_n(&signal->stepAndOwner, __ATOMIC_ACQUIRE);
ready = getOwnerDevice(loaded) == MoeLoadBalanceSingleLayerSignal::kCPU;
if (!ready && stop != nullptr && stop->load(std::memory_order_relaxed))
{
return false;
}
} while (!ready);
std::atomic_thread_fence(std::memory_order_acquire);
return true;
}

void moeSetSignalForGpuStageHost(MoeLoadBalanceSingleLayerSignal* signal, int64_t iterId, bool enableStatistic)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
#include "tensorrt_llm/common/config.h"
#include "tensorrt_llm/kernels/moe/loadBalance/moeLoadBalanceCommon.h"

#include <atomic>

TRTLLM_NAMESPACE_BEGIN

namespace kernels
Expand Down Expand Up @@ -110,8 +112,11 @@ void moeHierarchicalStatisticUpdate(MoeLoadBalanceMetaInfo metaInfo, MoeLoadBala
// @param tokenCount: the token count to compute the route
// @param offsetByEpRank: whether to offset the round robin position by epRank
// @param stream: the CUDA stream to be used
// @param singleReplicaPerExpert: whether every expert has a single replica, though empty spare slots may make the slots
// outnumber the experts. Selects the direct expert-to-slot kernel.
void moeComputeRouteDevice(MoeLoadBalanceMetaInfo metaInfo, MoePlacementInfo placementInfo,
int* const tokenSelectedExperts, int* tokenRoutedSlotIds, int tokenCount, bool offsetByEpRank, cudaStream_t stream);
int* const tokenSelectedExperts, int* tokenRoutedSlotIds, int tokenCount, bool offsetByEpRank, cudaStream_t stream,
bool singleReplicaPerExpert = false);

// @brief wait for the signal from gpu to cpu on host
//
Expand All @@ -120,7 +125,9 @@ void moeComputeRouteDevice(MoeLoadBalanceMetaInfo metaInfo, MoePlacementInfo pla
// After this function is called, functions for weight update can be called.
//
// @param signal: the signal
void moeWaitSignalForCpuStageHost(MoeLoadBalanceSingleLayerSignal* signal);
// @param stop: optional flag that ends the wait once set
// @return: false if the wait was stopped
bool moeWaitSignalForCpuStageHost(MoeLoadBalanceSingleLayerSignal* signal, std::atomic<bool> const* stop = nullptr);

// @brief set the signal for gpu stage on host
//
Expand Down
28 changes: 25 additions & 3 deletions cpp/tensorrt_llm/nanobind/runtime/moeBindings.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -95,12 +95,16 @@ void initMoeBindings(nb::module_& m)
.def("get_layer_id", &tr::SingleLayerMoeLoadBalancer::getLayerId,
"Get the layer id of the SingleLayerMoeLoadBalancer", nb::call_guard<nb::gil_scoped_release>())
.def("get_old_rank_expert_ids", &tr::SingleLayerMoeLoadBalancer::getOldRankExpertIds,
"Get the old rank expert ids of the SingleLayerMoeLoadBalancer", nb::call_guard<nb::gil_scoped_release>());
"Get the old rank expert ids of the SingleLayerMoeLoadBalancer", nb::call_guard<nb::gil_scoped_release>())
.def("get_unusable_local_slots", &tr::SingleLayerMoeLoadBalancer::getUnusableLocalSlots,
"Get the local slots that EPLB updates leave empty", nb::call_guard<nb::gil_scoped_release>());

// Bind MoeLoadBalancer class
nb::class_<tr::MoeLoadBalancer>(m, "MoeLoadBalancer")
.def(nb::init<int, int, int>(), nb::arg("ep_rank"), nb::arg("ep_size"), nb::arg("layer_updates_per_iter"),
"Initialize the MoeLoadBalancer with the specified expert parallel rank, size, and update frequency",
.def(nb::init<int, int, int, bool>(), nb::arg("ep_rank"), nb::arg("ep_size"), nb::arg("layer_updates_per_iter"),
nb::arg("fault_tolerance") = false,
"Initialize the MoeLoadBalancer with the specified expert parallel rank, size, update frequency and "
"whether Wide-EP fault tolerance is enabled",
nb::call_guard<nb::gil_scoped_release>())
.def("set_use_gpu_memcpy", &tr::MoeLoadBalancer::setUseGpuMemcpy, nb::arg("use_gpu_memcpy"),
"Set whether to use GPU memcpy for weight updates", nb::call_guard<nb::gil_scoped_release>())
Expand All @@ -120,11 +124,29 @@ void initMoeBindings(nb::module_& m)
.def("reconfigure_mask_only", &tr::MoeLoadBalancer::reconfigureMaskOnly, nb::arg("dead_ranks"),
"Reconfigure EPLB routing metadata so slots on dead EP ranks are unreachable",
nb::call_guard<nb::gil_scoped_release>())
.def("restore_placement", &tr::MoeLoadBalancer::restorePlacement, nb::arg("dead_ranks"),
nb::arg("layer_rank_expert_ids"),
"Under Wide-EP fault tolerance, set the failed EP ranks and install a placement per layer whose changed "
"local slots the caller loaded",
nb::call_guard<nb::gil_scoped_release>())
.def("abandon_iterations", &tr::MoeLoadBalancer::abandonIterations,
"Under Wide-EP fault tolerance, after a rank failure interrupted the forward, end the online EPLB work of "
"the iterations started so far and restart the statistics and the update plan",
nb::call_guard<nb::gil_scoped_release>())
.def("check_slots_writable", &tr::MoeLoadBalancer::checkSlotsWritable,
"Under Wide-EP fault tolerance, raise unless no iteration runs or is queued and the online EPLB worker is "
"done with the last one",
nb::call_guard<nb::gil_scoped_release>())
.def("shutdown", &tr::MoeLoadBalancer::shutdown, "Shutdown the load balancer and clean up resources",
nb::call_guard<nb::gil_scoped_release>());

m.def("is_host_accessible_device_memory_supported", &tr::HostAccessibleDeviceAllocator::isSupported,
"If current system support host accessible device memory");
m.def(
"host_accessible_device_memory_uses_gdrcopy",
[] { return tr::HostAccessibleDeviceAllocator::getInstance().usesGdrCopy(); },
"Whether host accessible device memory is device memory that GDRCopy maps for the host, which spare EPLB "
"slots of online EPLB need under Wide-EP fault tolerance; only while a load balancer exists");

// Bind do_replication function for testing
m.def("do_replication", &pyDoReplication, nb::arg("meta_info"), nb::arg("expert_load_factor"),
Expand Down
8 changes: 6 additions & 2 deletions cpp/tensorrt_llm/runtime/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,12 @@
include(FetchContent)

set(SRCS
bufferManagerNvls.cpp utils/debugUtils.cu ipcUtils.cpp
mcastDeviceMemory.cpp moeLoadBalancer/moeLoadBalancer.cpp
bufferManagerNvls.cpp
utils/debugUtils.cu
ipcUtils.cpp
mcastDeviceMemory.cpp
moeLoadBalancer/moeLoadBalancer.cpp
moeLoadBalancer/unbackedSlotMemory.cpp
ncclCommunicator.cpp)

include_directories(${API_INCLUDE_DIR}/tensorrt_llm/runtime)
Expand Down
Loading
Loading