Skip to content

[None][feat] Wide-EP fault tolerance: membership record, authority process and rendezvous store - #22675

Open
arusli-nv wants to merge 16 commits into
NVIDIA:feat/wide-ep-ft-mvpfrom
arusli-nv:wide-ep-ft/C-membership
Open

arusli-nv wants to merge 16 commits into
NVIDIA:feat/wide-ep-ft-mvpfrom
arusli-nv:wide-ep-ft/C-membership

Conversation

@arusli-nv

@arusli-nv arusli-nv commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

@coderabbitai summary

Description

PR 3 of 7 of Wide-EP fault tolerance, stacked on #22600 and #22674.
Every surviving rank must agree on which ranks are active, at which
generation, and where to resume, and a failed or hung process must never change that by itself.

  • Adds the membership authority: one CPU-only process on node 0 that decides the membership. Ranks report progress
    and waits to it; it detects crashes and hangs, drives each membership change and publishes the result.
  • Crash detection is a closed TCP liveness connection, so a crash is seen in milliseconds.
  • Hang rule: only the end of a wait-for chain that made no progress for the hang timeout (60 s) is removed, never the
    ranks that wait on it. A rank that its node supervisor reports paused is not hung.
  • Adds a signed rendezvous store and StoreGroup, an mpi4py-style communicator over the store whose exchanges reach
    only the active members.
  • New package only; the single change to existing code is a refactor of the DWDP store rendezvous.

What changes

Changes by file

New package tensorrt_llm/_torch/distributed/membership/:

  • records.py: MembershipRecord (generation, state, active rank mask, resume iteration and host-collective epoch,
    joining and given-up ranks, reason), RankReport, NodeReport (a node supervisor's given-up ranks and
    paused_mask, its ranks whose hang detector is paused and whose process is not stopped), and the timeouts
    (HANG_TIMEOUT_S = 60 s, reports every 0.5 s, heartbeats every 1 s).
  • authority.py: Authority, the decider. No I/O and an injected clock, so tests drive it with simulated ranks.
    • States: STARTING -> COMMITTED once every rank started; on a failure PROPOSE -> INSTALL -> COMMITTED. A failure
      during PROPOSE or INSTALL restarts PROPOSE with the larger failed set; a survivor replying FAILED, or the loss of
      rank 0, aborts the deployment.
    • A rank's paused_at.iteration is its next iteration (its executor's iteration count when it paused). INSTALL
      resumes every rank at the largest of these and from the latest paused epoch, so no rank repeats a step that
      another finished.
    • Hang rule: progress is the executor iteration plus the round flags and host-collective epochs that peers wait on,
      so a long step whose rounds advance is not hung. The hung ranks are the ends of the wait-for graph (strongly
      connected components that no wait leaves). Outside a change, a rank is not hung while the latest report of its
      node supervisor, received within half the hang timeout, lists it in paused_mask. This holds also once the
      rank's own reports stop, since the paused call may hold the GIL. A stopped rank is never in the mask. Once the
      rank leaves the pause, its stall starts no earlier than the latest node report that counted it as paused, so
      reports it could not send during the pause do not count against it.
  • authority_process.py, __main__.py: AuthorityProcess hosts the store (a TCPStore), accepts one liveness
    connection per rank and per node supervisor, publishes each record in the store and on every connection, and
    heartbeats idle connections. It admits a replacement only once the membership no longer lists its rank and no
    process of the rank is connected, and a first process only during the startup (_refusal). It keeps each supervisor's latest NodeReport with its arrival time, and logs once
    when a connected supervisor's report goes stale and once when it resumes. A supervisor that names ranks outside
    its node is dropped, so its ranks fail.
  • liveness.py: MembershipClient, the rank-side thread. It reports on a timer, hands each record to its owner, and
    ends the client when the authority is silent for half the hang timeout. A start that fails closes its connection,
    so the authority can admit a replacement. configure_connection sets TCP keepalive
    and TCP_NODELAY on both ends of every liveness connection, so each record goes out at once.
  • signed_store.py: SignedStore signs every value with HMAC-SHA256 and checks the tag before anyone parses or
    unpickles it; values over 4 MiB go in signed chunks. A hello token from the same secret is checked before a liveness
    connection changes the membership state. A missing secret or rendezvous address raises a ValueError that names
    the variable.
  • rendezvous.py: StoreGroup (allgather, bcast, allreduce, Barrier, Split, Split_type) over the store; an
    inactive member's value is None, and Split leaves inactive members out, as their colors are unknown.
  • Main's tensorrt_llm/_torch/modules/dwdp/rendezvous.py: the store allgather/Barrier of StoreDwdpGroup moves
    into a base class StoreComm, which StoreGroup reuses; DWDP keys and behavior are unchanged.
  • tensorrt_llm/_wide_ep_fault_tolerance.py: TLLM_FAULT_TOLERANCE_SHARED_SECRET (at least 32 bytes, hex) and
    TLLM_FAULT_TOLERANCE_RENDEZVOUS (the store's host:port).

Why a separate process: a busy rank 0 (GIL, long steps) must not stall membership decisions. Why the hang rule:
removing every rank that waits would take healthy ranks with a hung one; main's hang detector cannot tell the two
apart.

FT off stays main

  • Nothing outside the new package imports it.
  • The DWDP StoreComm extraction keeps DWDP's keys and calls; main's test_dwdp_rendezvous.py passes unchanged.

Stack and dependencies

This is PR 3 of 7. Each PR builds on the one before it:

  1. The enable_wide_ep_fault_tolerance field and a fault-tolerant MoE all-to-all ([None][feat] Wide-EP fault tolerance: enable_wide_ep_fault_tolerance and a fault-tolerant MoE all-to-all #22600).
  2. EPLB unbacked spare expert slots and expert restore into them.
  3. This PR: Membership record, authority process and rendezvous store.
  4. MNNVL memory over a changing membership, and MnnvlDist.
  5. Independent rank startup and the node supervisor.
  6. Executor failover and rejoin.
  7. Serving across a lost rank, multi-node launch, docs and end-to-end tests.
  • Stacks on PR 2 (extends its tensorrt_llm/_wide_ep_fault_tolerance.py), cut on PR 1 without the fix PRs. Through
    PR 2, needs [None][fix] Runtime and serving fixes: recompute replay, stream errors, frontend drain, VMM unmap #22507 to merge first; no other fix PR.
  • Until PRs 1-2 merge, this PR also shows their commits; the change of this PR is its last commit.
  • Product caller: rank 0 starts the authority and every rank connects in the rank startup of PR 5, which the field
    reaches in PR 6. The paused mask comes from PR 5's node supervisor, which reads each rank's pause word in /dev/shm;
    PR 6's hang detector writes the word. Until then the package runs in tests only.

Limits

  • Security: the store and the liveness listener bind only the rendezvous address (loopback for one node). Values and
    hello tokens are authenticated with the deployment's secret; traffic is not encrypted, and counters, deletes and
    replays of an older signed value are not protected. The network between nodes must be trusted, like main's
    TCPStore. Multi-node needs the liveness listener's ephemeral port reachable.
  • Node 0 hosts the authority: losing it ends the deployment.
  • A rank that truly hangs inside a call that pauses main's hang detector is not removed as hung; this matches main's
    detector. The ranks that wait on it stall until their own hang detector or the all-to-all timeout ends them
    (stated at Authority). A process that dies is still removed when its connection closes.
  • A node supervisor that stays connected but sends no report for over 30 s (half the hang timeout) ends its ranks'
    exemption; a rank of that node in a paused call that others wait on then counts as hung.
  • A hung rank is removed only after the 60 s hang timeout.

Evidence

At this ref:

  • This PR's changed tests on its own native build: 151 passed, 0 failed (job 9181945, the same change on PR 1's
    previous commit).
  • The unit tests this PR's files reach, run side by side with this change and on main: none fails only with this PR;
    36847 passed, 106 fail on main too (job 9183642, the same change on PR 1's previous commit). The 13 tests of
    test_dspark_eplb_config.py that failed there came from PR 1's previous commit; they pass at this ref (job
    9185800).
  • The tests this revision adds (test_split_leaves_out_inactive_members, test_store_client_requires_set_secret,
    test_rendezvous_store_requires_address, test_failed_client_start_closes_connection) each fail without their
    change and pass with it (job 9178351, at the stack's top).

On earlier code of the stack (the same membership logic; the job names the code):

  • Detect and agree (kill -> every survivor stopped), DeepSeek-R1 on 16 GPUs: node loss 88 ms (68-104), n = 5 (jobs
    9135191, 9135765, 9135779, 9135811, 9139604, code ff4a6bf); rank loss 13 ms (10-17), n = 2 (jobs 9043734,
    9059045, code fa51bbe). Commit (restore done -> COMMITTED): 119 ms (114-127), n = 5, node loss, same jobs.
  • Rejoin (authority window, propose -> installed): node loss 0.37 s (0.28-0.51), n = 5 (same node-loss jobs); rank
    loss 0.17 s (0.14-0.20), n = 2 (jobs 9043734, 9059045, code fa51bbe).
  • Hang (SIGSTOP of one rank): detected after 0.47 s, agreed after 60.53 s (60 s hang timeout), committed at 62.23 s,
    within the 65 s bound; n = 1 (job 9120386, code cbe98d2). A smoke run on the same code: 0.43 s, 60.52 s, 61.90 s
    (job 9119175).
  • test_authority_aborts_only_at_end_of_rank_0_pipe fails without the end-of-pipe rule and passes with it
    (job 9134161, the stack's top at the time).
  • Paused rule from the node reports: test_paused_rank_is_not_hung, test_node_report_keeps_paused_rank,
    test_paused_rank_kept_only_while_node_report_fresh and the foreign paused rank case of
    test_node_supervisor_loss_fails_its_ranks each fail with the stack's product files before the hang-rule change
    and pass with them, also in three parallel runs (job 9148832).

Size: +5085/-29 in 14 files (product +2612/-29, tests +2473).

Test Coverage

All CPU (cpu_only); the process tests use real sockets on loopback.

  • tests/unittest/_torch/distributed/test_membership.py (new):
    • Crash and hang: test_closed_connection_removes_the_rank, test_only_wait_chain_end_is_hung,
      test_stopped_process_is_hung_anywhere, test_delivered_rounds_are_not_a_hang,
      test_progressing_install_is_not_hung, test_paused_rank_is_not_hung (kept while its node supervisor reports
      it paused, also when its own reports stop; removed if not paused), test_node_report_keeps_paused_rank (over the
      authority process: kept while reported paused, removed once reported armed), test_late_rank_after_idle_not_hung,
      test_hang_clock_restarts_at_commit, test_hang_detection_dated_at_stall_start.
    • Transactions: test_commit_waits_for_every_install, test_second_failure_in_each_phase_commits,
      test_hang_during_transaction_restarts_it, test_failed_install_or_rank0_loss_aborts,
      test_second_failure_during_failover_install.
    • Startup: test_startup_rank_failure_aborts_at_once, test_startup_failure_aborts_at_rendezvous_timeout,
      test_startup_collective_hang_aborts, test_startup_without_node_supervisor_aborts.
    • Replacements and node supervisors: test_replacement_refused_until_rank_removed,
      test_foreign_or_second_replacement_refused, test_replacements_added_in_one_transaction,
      test_stalled_joiner_left_out_early, test_frozen_survivor_removed_before_joiner,
      test_node_supervisor_loss_fails_its_ranks (a closed connection, or a report that names a given-up or paused
      rank of another node), test_lost_node_rejoins_in_one_change,
      test_second_node_supervisor_refused.
    • Process and client: test_authority_survives_misbehaving_peer, test_authority_heartbeats_keep_ranks_connected,
      test_liveness_connections_configured (TCP_NODELAY and keepalive on both ends),
      test_rank_ends_on_silent_authority, test_authority_failure_aborts_with_reason.
    • authority_process.main and the store: test_authority_process_runs_until_rank_0_ends (started as rank 0 starts
      it, ends as rank 0 does), test_authority_aborts_only_at_end_of_rank_0_pipe,
      test_stall_published_in_unix_milliseconds, test_rendezvous_store_signs_with_shared_secret,
      test_rendezvous_store_requires_address, test_failed_client_start_closes_connection.
    • test_random_interleavings_keep_invariants (protocol invariants after every tick of random failures, joins and
      late replies).
  • tests/unittest/_torch/distributed/test_membership_hang_rule.py (new): every crash and stop pattern of small groups
    (test_only_chain_ends_are_hung), every wait graph of 2 to 4 ranks and random graphs against a reachability model
    (test_chain_end_matches_reachability, test_chain_ends_match_reachability_on_random_graphs),
    test_hang_timer_restarts_after_stall_ends, and test_paused_rank_kept_only_while_node_report_fresh (a paused
    rank is kept while its node report is fresh, and hung one hang timeout after its last report once that report is
    older than half the hang timeout), and test_hang_timer_starts_when_pause_ends (a rank whose reports stopped
    during a pause is kept when they resume after it, and hung one hang timeout after the pause if it stays stuck).
  • tests/unittest/_torch/distributed/test_store_group.py (new): collectives over the active members
    (test_allreduce_over_active_members, test_bcast_over_active_members), Split (also with an inactive member,
    test_split_leaves_out_inactive_members) and Split_type, and the signed store
    (test_forged_store_value_is_never_unpickled, test_forged_chunks_rejected, test_store_client_requires_ranks_secret,
    test_store_client_requires_set_secret, test_value_over_store_server_limit).
  • Main's tests/unittest/_torch/modules/dwdp/test_dwdp_rendezvous.py covers the StoreComm extraction.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

…and a fault-tolerant MoE all-to-all

PR 1 of 7 of Wide-EP fault tolerance: a wide expert-parallel deployment that keeps serving on fewer GPUs when a rank
fails, and takes the rank back when its replacement starts.

- Add the prototype field TorchLlmArgs.enable_wide_ep_fault_tolerance (default off).
- With the field on, the NVLink one-sided all-to-all reads its rank mask from GPU memory, so captured CUDA graphs
  follow a membership change without recapture.
- The host can abort the all-to-all waits on a peer; a long wait records which peer it waits on; no rank reads the
  rows of a round that a failure cut short.
- Remove main's earlier fault-tolerant MoE all-to-all (the host-side AlltoAll watchdog and wide_ep_ft.py, selected by
  TLLM_FAULT_TOLERANCE_MODE=1) with its knobs.
- With the field off, the all-to-all runs main's kernels and main's host path.

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
…ests' llm_args

ModelLoader reads llm_args.enable_wide_ep_fault_tolerance when it builds the model, so the
SimpleNamespace llm_args of the MX and GMS loader tests set it like their other fields.

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
Bind each MoE layer with the EP rank and size of its own communication, and refuse layers that disagree, drafter
layers included, and execution states of another EP size. Reject the configurations the arguments alone rule out at
validation, use the built-in model under modeling_v2 auto, and name the actual cause when a layer cannot be bound.
Forward the field to drafter model configs. Keep an abort request that arrives during rearm, refuse a mask install
while a layer is between its dispatch and its combine, and document the recovery order. Check the abort state's
pinning once, at construction, and refuse Confidential Computing. Warn at every MoE communication selection when a
removed all-to-all watchdog setting or TLLM_FAULT_TOLERANCE_MODE=1 is set. Describe the field's prerequisites, fix
stale comments and coding-guideline nits, bound the barriers of the multi-GPU tests, and test the fence timeout
without an abort state, the low-precision combine of an aborted round and an aborted combine.

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
…sts' model config

The drafter's model config now forwards enable_wide_ep_fault_tolerance from the target's, so the tests' model config stub carries the field, as a real ModelConfig does.

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
…o them

PR 2 of 7 of Wide-EP fault tolerance. After a rank fails, the survivors must serve its experts on the CUDA
graphs they already captured, without paying GPU memory for that while healthy.

- Adds unbacked spare EPLB slots: each rank's slot range is mapped at a fixed address, and spare slots map one shared
  scratch chunk until a restore backs them with memory of their own.
- Adds the restore: every surviving rank plans the same new placement, backs the spare slots the plan fills, loads
  the lost experts from host weights, and installs the placement.
- The MoE GEMM autotuner profiles only the active slots, so spare slots add no tuning workspace and change no tactic.
- A MoE backend or quantization method that cannot run with spare slots is turned down at selection, with the reason.
- Nothing creates spare slots before PR 6; with FT off, placement, routing, weight loading and tuning take main's paths.

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
…ocess and rendezvous store

PR 3 of 7 of Wide-EP fault tolerance. Every surviving rank must agree on which ranks are active, at which
generation, and where to resume, and a failed or hung process must never change that by itself.

- Adds the membership authority: one CPU-only process on node 0 that decides the membership. Ranks report progress
  and waits to it; it detects crashes and hangs, drives each membership change and publishes the result.
- Crash detection is a closed TCP liveness connection, so a crash is seen in milliseconds.
- Hang rule: only the end of a wait-for chain that made no progress for the hang timeout (60 s) is removed, never the
  ranks that wait on it. A rank that its node supervisor reports paused is not hung.
- Adds a signed rendezvous store and StoreGroup, an mpi4py-style communicator over the store whose exchanges reach
  only the active members.
- New package only; the single change to existing code is a refactor of the DWDP store rendezvous.

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
@arusli-nv

Copy link
Copy Markdown
Collaborator Author

/bot run

@arusli-nv

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86323 [ run ] triggered by Bot. Commit: 1c08963 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86323 [ run ] completed with state SUCCESS. Commit: 1c08963
/LLM/main/L0_MergeRequest_PR pipeline #71523 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@arusli-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86365 [ run ] triggered by Bot. Commit: 1c08963 Link to invocation

A rejoin restore can overwrite occupied slots, and a newer membership change can stop it before it installs its placement. A later plan that kept such a slot's old expert skipped it and then cleared the invalid slots, leaving the stopped restore's weights there. current_placement() returns the installed placement with this rank's invalid slots set to -1; an active rank's row of the plan's current placement is that view, so the plan loads those slots again on every rank alike.

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
…e wrote

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86365 [ run ] completed with state SUCCESS. Commit: 1c08963
/LLM/main/L0_MergeRequest_PR pipeline #71564 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@arusli-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86405 [ run ] triggered by Bot. Commit: df80eca Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86405 [ run ] completed with state SUCCESS. Commit: df80eca
/LLM/main/L0_MergeRequest_PR pipeline #71601 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@arusli-nv

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "Only failures are Kimi K3 modeling_v2 tests from the base branch (missing _fp4_mla_generation_num_blocks_device import, fixed on main in #22545); this PR does not touch those files."

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86435 [ skip ] triggered by Bot. Commit: df80eca Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86435 [ skip ] completed with state SUCCESS. Commit: df80eca
Skipping testing for commit df80eca

Link to invocation

@trtllm-agent-review trtllm-agent-review left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. Adds the Wide-EP fault-tolerance membership package: an Authority that decides membership and detects crashes and wait-chain hangs, its CPU-only process with a signed rendezvous store and liveness connections, the rank-side client, and StoreGroup over the store.

Reviewed at commit df80ecadcc (14 files, +5,085/−29).

This PR is stacked on #22674: this review covers its 3 own commit(s) on top of #22674, which is reviewed in its own PR.

Scope: this PR's own commits on top of #22674's head, be180d2..df80eca (1650f13, 1c08963 and the merge df80eca), 14 files.

Most possible blockers are reached once PR 5 or PR 6 adds the first product caller; they are raised here because the contract they concern is defined in this PR.

Request changes: 3 Blockers need fixing before merge.

Blockers (3)

  • (B1: Correctness) The hang rule drops a healthy replacement after the 20 s admission limit when a survivor stalls during the change that admits it. (authority.py:547)
  • (N3: Ineffective new test) test_unconnected_replacement_join_request_dropped cannot fail: its unconnected replacement never reports joining_ready, so it is never admitted whether or not the request is dropped. (test_membership.py:493)
  • (B3: CI breakage) The authority subprocess gets only 60 s to import tensorrt_llm._torch and write its address, about a 2x margin on 2-core CPU-only pods, so the CPU shard can flake. (test_membership.py:925)

Possible blockers (17)

  • (B1: Correctness) A normal shutdown is published as ABORTED (authority exit 1) when rank 0 exits or stops its client right after report(shutdown=True). (authority_process.py:193)
  • (B2: Crash, hang, race or leak) After SHUTDOWN the authority waits without a bound while any active rank stays connected and rank 0's stdin is open, which deadlocks a rank 0 that waits for its exit status. (authority_process.py:227)
  • (B2: Crash, hang, race or leak) MembershipClient waits with select.select, which raises ValueError for descriptors >= 1024, so a client started in a busy rank process fails and the rank is removed or the startup aborts. (liveness.py:375)
  • (B2: Crash, hang, race or leak) A second MembershipClient.stop() closes whatever descriptor reuses the old wake-pipe number and then raises TypeError. (liveness.py:293)
  • (B2: Crash, hang, race or leak) bind() or stop() called from a client callback deadlocks or raises, and a client failure before bind() never reaches the new on_error. (liveness.py:304)
  • (B2: Crash, hang, race or leak) A missing node supervisor is caught only if the startup takes over 60 s; a faster startup commits without it, and its idle rank 0 is later removed as hung. (authority_process.py:246)
  • (B2: Crash, hang, race or leak) python -m tensorrt_llm._torch.distributed.membership initializes MPI at import unless MPI4PY_RC_INITIALIZE=false and TRT_LLM_NO_LIB_INIT=1 are set; only the test sets them. (authority_process.py:605)
  • (B2: Crash, hang, race or leak) A starting rank that others wait on is failed after 600 s without startup progress, and weight loading does not count as progress, so a 10-minute load skew aborts the startup. (records.py:57)
  • (B1: Correctness) StoreGroup keeps its last call's keys in the store, so a new group under a reused name returns a peer's stale value at once and reports wrong waiting masks. (rendezvous.py:229)
  • (B2: Crash, hang, race or leak) Subgroup names come from a per-rank counter, so after subgroup() on some members, or a Split that raised on some ranks, the next Split's members wait on different keys. (rendezvous.py:181)
  • (B2: Crash, hang, race or leak) When the store server stops answering (authority stopped, node 0 frozen), the 0.5 s timed wait blocks in C++, so on_wait never runs and ranks stay in the collective. (rendezvous.py:76)
  • (B6: Security) StoreGroup unpickles whatever its store returns and is typed for a plain torch Store, so nothing enforces the SignedStore check that keeps forged pickles out. (rendezvous.py:93)
  • (B2: Crash, hang, race or leak) StoreGroup defaults on_wait to a no-op and gpus_per_node to 1, so an omitted argument means a silent unbounded wait or one-rank node groups. (rendezvous.py:97)
  • (B2: Crash, hang, race or leak) On EMFILE with no connection waiting for its hello, _accept leaves the connection queued, so the level-triggered listener spins the authority at 100% CPU. (authority_process.py:379)
  • (B1: Correctness) STALLED_SINCE_KEY is documented as empty while nothing stalls, but it is never written before the first stall, so a reader's get() blocks until the store timeout. (authority_process.py:145)
  • (B2: Crash, hang, race or leak) start() leaves the first record queued after setting self.record, so a bind() before the thread's first delivery gets that record twice. (liveness.py:274)
  • (B1: Correctness) Admitting a replacement bumps its incarnation without restamping the published record, so NodeViews show the rank connected with the old incarnation until the next change. (authority_process.py:456)
Nice to have (11): 7 inline, 4 below
  • (N1: Architecture and design) The generic StoreComm base lives in DWDP's module, so the membership package and the CPU-only authority import the DWDP package. (rendezvous.py:24)
  • (N1: Architecture and design) The CPU-only authority loads the LLM API, executor and DWDP stack (about 148 modules) through tensorrt_llm._torch, an eager package init and this llmapi import. (records.py:29)
  • (N1: Architecture and design) StoreGroup uses mpi4py names with different semantics: rank is global, Get_size() counts inactive members, MPI.UNDEFINED is a color, allreduce uses builtin sum. (rendezvous.py:121)
  • (N1: Architecture and design) StoreGroup and the StoreComm extraction land without a caller, and all exports 39 names, including timing constants, store keys and an affinity helper. (__init__.py:66)
  • (N1: Architecture and design) The hello token always comes from the environment, while SignedStore and open_store accept an explicit secret, so the two credentials can come from different secrets. (liveness.py:143)
  • (N5: Code tidiness) Several docstrings and comments misstate the code, and some code is dead. (rendezvous.py:210)
  • (N6: Duplication and reuse) The stale-report rule, the affinity opt-out gate and MAX_RANKS each live in three or more places, kept in line only by comments. (records.py:49)
  • [Nice to have] (R5: Input and config validation) Unsupported settings are accepted silently, or rejected late without naming the setting:
    • All-to-all timeout. records.py:36 states 'The all-to-all kernels must wait at least twice the hang timeout', but nothing enforces it. With fault tolerance on and TRTLLM_MOE_COMM_TIMEOUT_SEC below 120 s (an accepted value), wait_round_flag<HONOR_ABORT> still ends in trap; at that timeout (moeAlltoAllKernels.cu:377-425). One hung rank then crashes every survivor before the authority (60 s, up to 120 s during a change) can remove it. Validate this at fault-tolerance launch or when the authority starts, naming both settings.
    • Rendezvous address. rendezvous_host_port (rendezvous.py:61-62) validates nothing. ':29500' binds every interface, 'node0' fails with an unpacking error, '[::1]:29500' keeps the brackets, and 'host:0' is accepted. Raise a ValueError naming TLLM_FAULT_TOLERANCE_RENDEZVOUS.
    • Launcher arguments. main() writes the address to --address-fd before Authority(args.world_size) validates the world size, and it never validates --gpus-per-node (rank // self._gpus_per_node at 253). Validate both with parser.error before binding.
    • Secret check order. rendezvous_store() connects before it parses the secret. A missing secret with the store unreachable gives a 60 s DistNetworkError instead of the documented ValueError.
    • Store values. SignedStore.set coerces with bytes(value), which turns an int n into n zero bytes. Require bytes.
    • Own rank. StoreGroup never checks that its own rank is in active. Raise in __init__.
  • [Nice to have] (N7: Robustness hardening) Failure paths and diagnostics to harden:
    • Report sends (liveness.py:332). They use a blocking sendall with no timeout, on the thread that runs the 30 s silence check. With a frozen authority node and frequent reports, detection takes about 60 s (TCP_USER_TIMEOUT). Set the remaining silence budget as the socket timeout before each send.
    • Startup loss between ticks (authority.py:410). _connected_once is updated only at tick time. A first process that is admitted and then dropped before the next decision (for example, its first-record send fails) makes the startup wait 600 s, and it then aborts with 'did not reach the rendezvous'. Record admission when the hello is accepted.
    • Missing transaction (authority_process.py:194). The abort for rank 0's process ending skips _record_transaction, so TRANSACTION_KEY never shows the final aborted change.
    • Hang warnings (authority.py:609). A rank that never connected is logged as 'hung', with made-up last reports, just before the 'did not reach the rendezvous' abort. Each hung rank also gets its own warning describing every active rank, about 59 kB each at 256 ranks.
    • Refusals (liveness.py:268-271). A refused process only sees 'refused this process of rank N'. Send the reason as a line before closing.
    • Log level (authority_process.py:618-622). The authority sets no log level, so at the default ('error') its warnings and infos are hidden: refusals, a supervisor that left, hung ranks, records. Add a --log-level option or default to info.
    • c10d warnings (rendezvous.py:258). Each 0.5 s timed TCPStore.wait prints two c10d WARNING lines, about 4 lines/s per waiting rank. 256 ranks waiting in a startup exchange print hundreds of thousands. Poll with check() and a sleep instead.
  • [Nice to have] (N3: Missing or ineffective tests) Tests that would catch regressions in the core rules:
    • Rules no test pins. Each of these passes all 151 tests when removed:

      • the replaced-INSTALL epoch rule (authority.py:379-380);
      • the resume-iteration floor (365);
      • the commit clearing _joining (387);
      • the at-fork close of liveness sockets (liveness.py:108);
      • the key binding of the HMAC tag (signed_store.py:149);
      • the '1' value of the affinity gate (records.py:71);
      • the sibling-wait condition _reports_advance (authority.py:442).

      Suggested coverage, in the same order:

      • a Cluster test that moves paused_at.epoch to a replaced INSTALL's epoch;
      • assert record.joining_mask == 0 on COMMITTED records in the random-interleavings invariants;
      • a forked-child EOF test;
      • store.set("forged/0/1", store.get("own")) raising UnsignedValueError;
      • a mocked unit test of the affinity helper;
      • a test with a stopped sibling replacement.
    • test_authority_survives_misbehaving_peer (test_membership.py:1495) never parses its invalid line: the patched send drops the connection first. So the guards against bad peers are all untested: the invalid-report close (authority_process.py:419-422), the hello timeout, the pending-hello cap, the 256-byte hello limit and the unsent-bytes cap. Parametrize it over an admitted peer that sends b"not a report\n", a peer with no hello, an over-long hello, and a peer that reads nothing.

    • Test economy. Together these save about 30 s per CPU run:

      • test_only_chain_ends_are_hung[5] (test_membership_hang_rule.py:226) runs 13,487 cases (about 10 s), and worlds 3-4 catch everything it catches.
      • Several process-level waits are sized like production timeouts: hang_timeout_s=5.0 at 823, the 2 s silence timeouts at 1030 and 1577, the 3 s loop at 971.
      • The four hand-written Authority.tick wrappers (1281, 1321, 1374, 1434) can be one fixture.
      • ids=lambda s: s.name would make the MembershipState parametrize ids readable.
  • [Nice to have] (R3: Repository rules) Written repository rules this change breaks (AGENTS.md: 'Read and follow CODING_GUIDELINES.md for ALL code changes'):
    • CODING_GUIDELINES.md:664, 'Always annotate functions'. Unannotated parameters: __exit__(self, *exc_info), report(self, **changes) and _call(..., *args) (liveness.py:77, 309, 413). The new test helpers are unannotated too, including InMemoryStore, whose signatures stand in for TCPStore. StoreComm also uses typing.List (dwdp/rendezvous.py:126, 142), against ':674 Prefer using the built-in types list...'.
    • CODING_GUIDELINES.md:520, 'Avoid shadowing variables declared in an outer scope.' states = [] at test_membership.py:1174 and 1226 shadows def states(cluster) (171).
    • CODING_GUIDELINES.md:590, 'Externally called functions should have docstrings, and their arguments should be documented. Class initializer arguments especially should be documented.' MembershipClient.stop() (liveness.py:284) has no docstring, and StoreComm.__init__ documents none of its arguments.
    • tests/unittest/AGENTS.md:41-43, '... clean up owned threads, processes and connections.' The _NodeSupervisor connections (test_membership.py:968, 1070, 1182-1183, ...) are never closed. _leave (812-815) skips stop() for clients whose thread ended, which leaks about 38 pipe descriptors per run.
What this review checked
  • The DWDP StoreComm extraction keeps DWDP's keys (dwdp///), call order and freed-group errors; main's test_dwdp_rendezvous.py passes on the old and the new file and runs pre-merge (l0_cpu.yml:203).
  • The hang rule's chain ends match a reachability model on every wait graph of 2-4 ranks, and the pause exemption, the stall start after a pause and the node-report freshness rule each fail their tests when removed.
  • Malformed hellos and lines from peers without the token (empty, wrong field counts, non-UTF-8, huge integers, 70 KB lines) only drop that peer, and StoreGroup's key deletion is safe when every member uses the same active set.
  • Not covered: No GPU or multi-node run: the changed modules ran only in a CPU harness with stubbed TensorRT-LLM imports, on torch 2.11 (the pin is 2.14).
  • Not covered: The PR 5 and PR 6 callers (rank startup, node supervisor, executor failover) are not available, so how they shut down, create the client and name groups is inferred.
  • Not covered: The commits of #22674 and #22600 underneath this PR.

Automated review by trtllm-agent-review. If a point is wrong, reply in its thread. After you push fixes or reply, the bot re-checks the Blockers within a few hours.

waited_on = 0
if state in expected:
waited_on = rank_mask(
(r for r in active if not self._replied(reports.get(r), expected[state])),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Blocker] (B1: Correctness) The hang rule and the change logic disagree on when a joiner has answered, so a healthy replacement is removed as hung:

  • PROPOSE. waited_on here uses _replied, which needs an explicit READY. _advance_change (line 343) uses _rank_replied, which counts joining_ready as READY, as RankReport.joining_ready documents ("It counts as its READY to the change that adds it", records.py:241-242). So a joiner that sends only joining_ready is waited on, waits on nobody (a chain end), and stalled_joiner (602-603) gives it min(timeout_s, admission_timeout_s) = 20 s.
  • INSTALL. A survivor that freezes while its last report says waiting_for=<joiner> keeps that wait counted for half the hang timeout (30 s, _reports_advance). The joiner that already replied INSTALLED and idles is then the waited-on chain end, and the same 20 s limit removes it.

Simulated with the default timeouts (world 4, rank 3 replaced):

  • Survivor 1 stuck in the admission's PROPOSE: the joiner is dropped about 21 s after the PROPOSE, and the stuck survivor 40 s later. With an explicit READY, the joiner is kept and committed.
  • Frozen-survivor INSTALL case: the joiner goes at about 21 s, the frozen survivor at about 82 s.

The dropped replacement's client ends, so it has to be restarted and rejoin in yet another change. test_frozen_survivor_removed_before_joiner passes only because its joiner waits back on rank 2.

Suggested fix: build waited_on with self._rank_replied(r, reports, expected[state]), and give the admission limit only to a joiner that has not given the phase's reply:

stalled_joiner = (r in self._joining and not waits.get(r)
                  and not self._rank_replied(r, reports, expected.get(state)))

Add Cluster tests for both cases (a joiner that sends only joining_ready while a survivor hangs; a survivor that freezes waiting on a joiner that replied INSTALLED), each expecting the joiner to commit.

decide_s = now_s + REPORT_INTERVAL_S
self._send_heartbeats(time.monotonic())
summary = self._authority.last_transaction
if self._parent_ended_s is not None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Possible blocker] (B1: Correctness) A clean shutdown ends as ABORTED unless rank 0 waits for the SHUTDOWN record before it stops its client and exits, and nothing documents that wait. There are three paths:

  • Here, rank 0's pipe EOF wins over a shutdown report the authority already holds. If the report and the EOF are read in the same poll, abort("rank 0's process ended") runs before tick() can decide SHUTDOWN.
  • If rank 0's liveness connection closes first, _reports() (558-559) no longer has its report, and tick() publishes ABORTED with 'rank 0 failed'.
  • In MembershipClient._serve the loop checks _stop first (liveness.py:363). A report(shutdown=True) made while _send_report is sending the previous copy is never sent once stop() follows.

Reproduced over loopback:

  • report-then-exit with a busy decision loop gave ABORTED 10/10;
  • report-then-stop() lost the shutdown report whenever the send overlapped.

Every shutdown test waits for SHUTDOWN first, so CI does not see this. The rank-0 shutdown path in PR 5/6 inherits it unless it waits too. Every rank then gets ABORTED and main() returns 1.

Suggested fix:

  • In _serve, decide rank 0's latest report before the parent-ended abort: if it carries shutdown=True, let tick() publish SHUTDOWN.
  • In stop(), send the latest report once more if it changed since the last send.
  • Document the handshake on RankReport.shutdown.

# The store lives in this process, so keep it until every active rank has read the
# record and left, for at most a hang timeout after an abort or rank 0's exit.
aborted_s = time.monotonic() if record.state == MembershipState.ABORTED else None
while self._connected() & record.active_mask:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Possible blocker] (B2: Crash, hang, race or leak) For SHUTDOWN, aborted_s is None, so the only bound on this loop is _parent_ended_s. That stays None until rank 0's pipe ends. So:

  • If one active rank stays connected after SHUTDOWN, the authority serves for as long as rank 0's stdin is open. That rank can be hung, SIGSTOPped, or simply never call stop(); a SHUTDOWN record keeps it a member.
  • main() ignores SIGINT and SIGTERM (645-646).
  • A rank 0 that waits for the authority's exit status before closing its stdin therefore hangs.

test_authority_process_runs_until_rank_0_ends plays rank 0 exactly that way: clients[0].stop(), then process.wait(...) with stdin still open. It works only because the world size is 1.

Reproduced with --world-size=2 and rank 1 connected: still serving after 10 hang timeouts with stdin open; it exited one hang timeout after stdin closed.

Suggested fix: also bound the SHUTDOWN linger from the SHUTDOWN publish, for example one hang timeout (or a longer limit while rank 0 lives). Document in main() that rank 0 closes stdin before it waits for the exit status. Test: world 2, rank 1 stays connected, stdin open; run() returns.

return
listed = member
watched = [self._sock, self._wake_read]
readable, _, _ = select.select(watched, [], [], max(0.0, tick_s - time.monotonic()))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Possible blocker] (B2: Crash, hang, race or leak) select.select cannot watch descriptors at or above FD_SETSIZE (1024): CPython raises ValueError: filedescriptor out of range in select(). _run (338-340) treats that as a lost authority and closes the liveness connection. The authority then removes the rank, or aborts the startup.

When it happens:

  • The wake pipe is created in __init__ and the socket in start(), so both get numbers above 1024 once the process already holds about 1020 descriptors.
  • A wide-EP rank can easily hold that many (CUDA, NCCL/UCX/NIXL sockets, the imported MNNVL handles of every peer).

Reproduced with 1100 extra descriptors open: 'Deployment aborted: ranks [0] failed during the startup'. The authority side already uses selectors (authority_process.py:138).

Suggested fix: register self._sock and self._wake_read once in a selectors.DefaultSelector() (epoll), close it in stop(), and adapt test_late_wakeup_keeps_live_authority, which patches liveness.select. A test that opens 1100 descriptors before start() and expects COMMITTED fails today.

self._sock.close()
# A report() racing this stop must not write to a closed or reused descriptor.
with self._lock:
os.close(self._wake_read)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Possible blocker] (B2: Crash, hang, race or leak) stop() closes both pipe ends on every call but only sets _wake_write = None. A second call, say once on an error path and again at teardown, does two things:

  • it runs os.close(self._wake_read) on a number the process may have given to another socket or file since;
  • it then raises TypeError on os.close(None).

Reproduced: after the first stop(), a new pipe got the freed numbers, and the second stop() closed it. If the reused descriptor is a store or NCCL socket, or rank 0's pipe to the authority (whose EOF aborts the deployment), the damage shows up far from this call.

The test helper _leave calls stop() only while the thread is alive, which hides this.

Suggested fix: make stop() idempotent: under _lock, return when it already ran, close each descriptor once and set both to None. Give it a docstring that says so.

len(self.membership),
timedelta(seconds=REPORT_INTERVAL_S),
)
self.rank = rank

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N1: Architecture and design) StoreGroup reads like an mpi4py communicator but differs in ways the docstrings only partly state:

  • rank is the global rank, while mpi4py's Comm.rank equals Get_rank(), so bcast(x, root=group.rank) raises IndexError in a subgroup.
  • Get_size() counts inactive members, whose values are None.
  • Split treats MPI.UNDEFINED as a color, while Split_type keeps inactive members.
  • allreduce(SUM) uses builtin sum: a TypeError for lists or strings, and an active None is silently dropped.
  • membership lists every member, not the active ones.

PR 4 already works around part of this with its own MembershipComm protocol. Stating the contract once, as a Protocol in _torch/distributed, and renaming rank to global_rank would keep the next caller from passing a StoreGroup to code written for mpi4py.

)
from .signed_store import SignedStore, UnsignedValueError, parse_shared_secret

__all__ = [

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N1: Architecture and design) Nothing outside the package uses it yet, and StoreGroup's first user is PR 4's MNNVL memory. So any mismatch PR 4 finds in StoreGroup's semantics means changing already-merged shared code, the DWDP base included.

Consider:

  • keeping the authority unit here: records, Authority, AuthorityProcess, MembershipClient, SignedStore and their tests;
  • moving StoreGroup, its tests and the StoreComm extraction to PR 4;
  • trimming __all__ to the names later PRs import (PR 4 uses StoreGroup, all_ranks, has_rank, next_epoch and rank_mask).

sock = socket.create_connection(address, timeout=timeout_s)
configure_connection(sock)
lines = Lines()
sock.sendall(lines.encode(f"{hello_token()} {announcement}".encode()))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N1: Architecture and design) hello_token() reads TLLM_FAULT_TOLERANCE_SHARED_SECRET from the environment (through shared_secret_digest), while SignedStore and open_store take an explicit secret. So a supervisor or rank that is given the secret explicitly signs its store values with it, and then fails at the hello: either with a ValueError naming a variable it was never told to set, or with a refusal when the environment holds another secret.

Either drop the secret parameter (the tests can monkeypatch.setenv), or pass the same secret into hello_token() and shared_secret_digest().

) -> "StoreGroup":
"""Return the members on this rank's node, supporting only the node split types.

The subgroup orders its members by global rank; ``key`` and ``info`` are ignored.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N5: Code tidiness) A few comments and docstrings say what the code does not do:

  • rendezvous.py:210 says Split_type orders by global rank; it keeps the parent's order.
  • rendezvous.py:108-109 says on_wait gets 0 'once the call has them'; it also gets 0 when the wait raised (227-228).
  • authority.py:117-120 (and the PR's Limits) say ranks waiting on a paused hung rank 'stall until their own hang detector or all-to-all timeout ends them'. But idle non-root ranks wait in the pause-wrapped request-count probe (request_utils.py:466-467), so neither fires and the stall is unbounded, as in main.
  • records.py:172 documents reason as 'Why a change left a rank out', but failovers are proposed with an empty reason (authority.py:291).
  • liveness.py:177 says timeout_s bounds 'the connection to the authority', but the call first waits 30 s for the liveness key and then up to the store timeout.
  • authority_process.py:482-483 says 'only a new given-up rank is decided at once', but any change is.

Dead code:

  • the ValueError at authority_process.py:409-410 is unreachable;
  • StoreGroup passes StoreComm a timeout it never uses (119);
  • StoreGroup redefines Barrier, which the base already provides (155-158).

HEARTBEAT_INTERVAL_S = 1.0
# The age at which a rank takes the authority as silent. It is half the hang timeout, as the
# authority takes a rank's reports as stale after half of its own hang timeout.
AUTHORITY_SILENCE_TIMEOUT_S = HANG_TIMEOUT_S / 2

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N6: Duplication and reuse) Three rules have several homes:

  • Stale reports. 'A report is stale after half the hang timeout' appears at authority.py:214 and 462 and authority_process.py:567, all from the instance's hang_timeout_s, and here from the module default. So Authority(hang_timeout_s=...) moves the authority's copies but not the client's 30 s silence timeout, and the 'no longer exempt' log can disagree with the decision.
  • Affinity gate. The TLLM_NUMA_AWARE_WORKER_AFFINITY in (None, "1") gate (records.py:71) is a third copy, after py_executor.py:2366 and configure_cpu_affinity. Unlike the latter, it skips TLLM_WORKER_AFFINITY_EXCLUDE_CPUS.
  • Rank limit. MAX_RANKS = 256 restates the kernels' kMaxRanks and EP_MASK_NUM_WORDS.

Suggested fix:

  • one Authority attribute such as report_stale_s;
  • a public no-GPU affinity helper next to configure_cpu_affinity;
  • a cpu_only test asserting MAX_RANKS == thop.MOE_A2A_MAX_RANKS.

@arusli-nv arusli-nv added the api-compatible Accepted LLM API contract change that is backwards-compatible label Nov 21, 2026
… leaves empty

A placement can leave an active slot empty (a restore stopped after it loaded a survivor's active slots), and the load balancer reports every empty slot as unusable. Releasing them all passed the active slot to the unbacked-slot memory, which refuses it, so nothing was released. Release only the unusable slots that are spare slots.

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
…nt leaves empty

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
…not reply

The authority counts a joining rank's joining_ready report as its READY when it decides that every rank replied, but the hang check listed the ranks still waited on without it, so a joiner at READY could be removed as hung.

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
select.select refuses file descriptors at or above FD_SETSIZE (1024); the membership client now waits through selectors.

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
… for a rank that started a round

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
A backend that declares supports_spare_slots keeps its slot-first expert weights in the spare-slot storage through loading, binds tuner_layout_kwargs into every tunable op and sizes runtime workspaces for every local slot. The fallback warning is logged once per requested backend and chosen class.

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
… guide

Signed-off-by: Andrew Rusli <arusli@nvidia.com>
@arusli-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86502 [ run ] triggered by Bot. Commit: 4df2ae4 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86502 [ run ] completed with state SUCCESS. Commit: 4df2ae4
/LLM/main/L0_MergeRequest_PR pipeline #71690 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@trtllm-agent-review trtllm-agent-review left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up check: 3 of the 3 Blockers of the earlier review are still open.

  • [Blocker] (B1: Correctness) The waited_on fix at line 547 resolves the PROPOSE case, but the INSTALL case is still open. stalled_joiner (line 602) still applies the 20 s admission limit to a joiner that already replied INSTALLED. Lines 561-564 keep a frozen survivor's last waiting_for=<joiner> for half the hang timeout. Replaying the test's Cluster with default timeouts (rank 3 replaced, survivor 1 freezes in INSTALL while waiting on the joiner, the joiner replied INSTALLED and idles) removes the healthy joiner 20 s after the freeze. It then has to restart and rejoin in another change. With and not self._rank_replied(r, reports, expected.get(state)) added to stalled_joiner, survivor 1 is removed at 60 s instead and the joiner stays. Please add that check, or an equivalent one, and a Cluster test where the joiner does not wait back on the frozen survivor and is expected to commit. test_frozen_survivor_removed_before_joiner passes only because its joiner waits back on rank 2. (authority.py:547)
  • [Blocker] (N3: Ineffective new test) test_unconnected_replacement_join_request_dropped (test_membership.py:512-521) is unchanged and still cannot fail. Its replacement reports RankReport(), so joining_ready is False and _start_change never admits it, whether or not _forget_joins drops the request. The test still passes with _forget_joins made a no-op, so it does not cover the rule it is named for. Please give the unconnected rank a stale RankReport(joining_ready=True) and assert that no published record makes rank 3 active. That version fails with the no-op and passes with the current code. (test_membership.py:493)
  • [Blocker] (B3: CI breakage) Still open: test_authority_process_runs_until_rank_0_ends still waits only TIMEOUT_S = 60 s (line 79, used at line 949) for the child to write its address. Before it can write it, python -m tensorrt_llm._torch.distributed.membership imports tensorrt_llm._torch (the LLM and executor stack) while pinned to one CPU; main() lifts the pin only at authority_process.py:623. The test runs in the CPU stage (l0_cpu.yml, unittest/_torch/distributed) on 2-core pods, so a slow fresh import fails both parametrizations and makes the CPU shard flaky. Please give the address wait its own, much larger startup bound, or keep the authority entry point from importing the tensorrt_llm._torch stack. (test_membership.py:925)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-compatible Accepted LLM API contract change that is backwards-compatible

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants