Skip to content

[None][fix] Fail pending and new RPC requests after the executor event loop crashes - #22690

Draft
lori-ren wants to merge 1 commit into
NVIDIA:mainfrom
lori-ren:fix/rpc-worker-event-loop-crash
Draft

lori-ren wants to merge 1 commit into
NVIDIA:mainfrom
lori-ren:fix/rpc-worker-event-loop-crash

Conversation

@lori-ren

@lori-ren lori-ren commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

@coderabbitai summary

Description

After an executor event-loop crash, pending RPC requests can wait indefinitely, including requests submitted after shutdown. The RPC worker now reads the shutdown state before polling, dispatches responses the engine already flushed, and fails the requests still pending. The delivery gate stays clear so the multi-rank crash watchdog can terminate stranded peers.

This PR follows up on the RPC hang reported during review of #22547.

Test Coverage

Ran test_rpc_worker_mixin.py, test_event_loop_error_broadcast.py, and test_rpc_worker.py::TestRpcWorkerTP1 on an NVIDIA A10. The new regressions failed on the base revision and passed with the fix. Moving the shutdown check after the poll reproduced a premature error for a request whose final response was already queued.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

…t loop crashes

Signed-off-by: Lori Ren <lorir@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant