Skip to content

[TRTLLM-17529][refactor] separate response completion from resource release - #22687

Draft
Funatiq wants to merge 1 commit into
NVIDIA:mainfrom
Funatiq:dev/feat/response_release
Draft

Funatiq wants to merge 1 commit into
NVIDIA:mainfrom
Funatiq:dev/feat/response_release

Conversation

@Funatiq

@Funatiq Funatiq commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Description

Native request completion also covers context transmission in progress, so final-response readiness cannot authorize allocation teardown. Introduce explicit scheduling, response, and release predicates while preserving the existing transfer owners, connector load deferral, and collective boundaries.

Publish responses before terminating their requests through one shared operation. When independent connector and transceiver completions race with a staged response, defer teardown until publication retains the routing. Retry early teardown after the last claim releases unless it actually completed, rather than skipping solely on the partial-reuse configuration.

Track allocation release and terminal teardown independently, reset allocation state for replay, and retain completed PP consensus while local response or DMA owners drain. This prevents duplicate frees after cancellation during recompute and avoids requesting another vote from already-released PP peers.

Add 27 CPU CI regression cases, including both dual-owner completion orders and the real two-rank PP termination handler with delayed publication.

@coderabbitai summary

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@Funatiq

Funatiq commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86350 [ run ] triggered by Bot. Commit: 940ebcc Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86350 [ run ] completed with state SUCCESS. Commit: 940ebcc
/LLM/main/L0_MergeRequest_PR pipeline #71548 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@Funatiq

Funatiq commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

Native request completion also covers context transmission in progress, so
final-response readiness cannot authorize allocation teardown. Introduce
explicit scheduling, response, and release predicates while preserving the
existing transfer owners, connector load deferral, and collective boundaries.

Publish responses before terminating their requests through one shared
operation. When independent connector and transceiver completions race with
a staged response, defer teardown until publication retains the routing.
Retry early teardown after the last claim releases unless it actually
completed, rather than skipping solely on the partial-reuse configuration.

Track allocation release and terminal teardown independently, reset allocation
state for replay, and retain completed PP consensus while local response or
DMA owners drain. This prevents duplicate frees after cancellation during
recompute and avoids requesting another vote from already-released PP peers.

Add 27 CPU CI regression cases, including both dual-owner completion orders
and the real two-rank PP termination handler with delayed publication.

Validation: 1624 focused CPU tests passed (108 skipped, 65 deselected with the
previously established connector baseline exclusions); 14 separately selected
recompute tests passed. A100 Qwen validation passed 16 generation scenarios,
cancellation and subsequent generation in both overlap modes, and four real
IPC/RPC worker scenarios. Cross-GPU model and transfer validation remains
pending because this container exposes one GPU.

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

# Conflicts:
#	tensorrt_llm/_torch/pyexecutor/py_executor.py

# Conflicts:
#	tensorrt_llm/_torch/pyexecutor/py_executor.py
@Funatiq
Funatiq force-pushed the dev/feat/response_release branch from 940ebcc to 506cd28 Compare November 21, 2026 11:01
@Funatiq

Funatiq commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86524 [ run ] triggered by Bot. Commit: 506cd28 Link to invocation

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants