Skip to content

[TRTLLM-17528][refactor] Improve response handling and regression coverage - #22682

Open
Funatiq wants to merge 4 commits into
NVIDIA:mainfrom
Funatiq:dev/feat/response_2
Open

Funatiq wants to merge 4 commits into
NVIDIA:mainfrom
Funatiq:dev/feat/response_2

Conversation

@Funatiq

@Funatiq Funatiq commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Follow-up to #22346.

  • Preserves final-response counters: publishes current sampler metadata before responses from early context-transfer completion and cancellation between batches.
  • Makes additional-output snapshots cheaper: records per-output list lengths while sharing storage. Reads and serialization respect those boundaries, keeping later tokens out of earlier responses.
  • Removes request-only PyResult.Diff from response snapshots.
  • Strengthens regression coverage: tests appended beam logprobs, delayed pickle/cloudpickle serialization, and logits exclusion remaining stable after the request changes.

Dev Engineer Review

  • Shared snapshot lists (llm_request.py, +39/-6): Snapshots now share output lists and store per-output length bounds. Reads and pickling must truncate to those bounds. A missed bound path would expose later appends. Verify every output accessor and the pickle path.
  • Removed cached_tokens copy (py_executor.py): prepare_response replaces the explicit copy. Confirm that cached_tokens still reaches the terminal response.
  • Diff removal: Snapshots no longer create a fresh Diff. Confirm no consumer reads Diff from a snapshot.
  • CI status: Pipeline #71533 failed. Pipeline #71587 completed as UNSTABLE. The comments give no failing-test details. Current-head CI status is not available.

QA Engineer Review

  • Test files changed: tests/unittest/_torch/disaggregation/coordinator_harness.py, test_disagg_coordinator_transfers.py, test_disagg_idle_cancellation.py, and tests/unittest/_torch/pyexecutor/test_request_utils.py.
  • Test functions added or modified: A fast-completion transfer test. A between-batch cancellation test for streaming, LoRA, and speculative decoding. Beam-logprob history tests. Generation-logit snapshot stability tests. Serialization bound tests for additional outputs.
  • Covered behavior: Sampler metadata in early-transfer and cancellation responses. Draft and acceptance counters. Beam logprob accumulation. Snapshot isolation from later appends. Single release on cancellation.
  • Test-list membership: No test-db/ or qa/ list files changed. Whether these tests are listed in CI is not available.
  • Author-reported results: 186 transfer and idle-cancellation tests passed. A 2241-test CPU suite passed, with 114 skipped and 10 deselected. These results are not verified on the current head.
  • Coverage verdict: Needs follow-up. Confirm current-head CI and the test-list membership.

Per-File QA Perspective

  • coordinator.py: release_transfer publishes metadata for settled active requests. Verify responses after transfer completion include the current metadata.
  • llm_request.py: Output lists are bounded by recorded length. Verify serialized responses exclude outputs added after response creation.
  • py_executor.py: Cancelled disaggregated requests now use prepare_response(publish_metadata=True). Verify cancellation responses and deferred termination.
  • coordinator_harness.py: py_draft_tokens defaults to an empty list. No QA impact beyond test setup.
  • test_disagg_coordinator_transfers.py: Covers fast-completion metadata. Not verified in CI test lists.
  • test_disagg_idle_cancellation.py: Adds a streaming option and a between-batch cancellation regression. Not verified in CI test lists.
  • test_request_utils.py: Covers beam logprob history and response snapshot bounds. Not verified in CI test lists.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Publish current Python sampler progress before an early context-transfer completion creates its native result. Prepare between-batch cancellation responses through the same helper so their counters are published and their speculative metadata is snapshotted instead of reverting to empty defaults.

Preserve the native transfer state during response creation and the existing connector-save ordering. Add native serialized-result regressions for decoding iteration at early completion and current cancellation counters alongside an unchanged earlier response.

Validation: the new regressions reproduced stale iteration and draft state before the fix. All 186 transfer/idle-cancellation tests and the broader 2241-test CPU suite pass, with 114 skipped and 10 deselected.
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Keep request-only PyResult.Diff out of responses. Record per-output list lengths when snapshotting instead of copying all accumulated additional-output lists at every decoding step.

Bound both property reads and serialized state to the recorded prefixes so later context or generation chunks cannot leak into queued responses. Cover shared list ownership, independent output boundaries, initially empty outputs, and delayed pickle/cloudpickle round trips.

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Model the sampler second step with append_log_probs instead of replacing the history. Assert that beam histories grow and cumulative logprobs advance on the request while the first response retains only its original token logprobs and cumulative totals.

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Create a streaming response after two distinct logits chunks with exclusion enabled, using prepare_response without the latest-logits override. Append a third chunk and disable exclusion on the live request, then verify that the retained response still exposes the first chunk while the request exposes the third.

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
@Funatiq

Funatiq commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86334 [ run ] triggered by Bot. Commit: 6ce436a Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86334 [ run ] completed with state SUCCESS. Commit: 6ce436a
/LLM/main/L0_MergeRequest_PR pipeline #71533 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@Funatiq

Funatiq commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86389 [ run ] triggered by Bot. Commit: 6ce436a Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86389 [ run ] completed with state SUCCESS. Commit: 6ce436a
/LLM/main/L0_MergeRequest_PR pipeline #71587 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

  • Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
  • If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
    Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

@Funatiq
Funatiq marked this pull request as ready for review November 21, 2026 10:38
@Funatiq
Funatiq requested review from a team as code owners November 21, 2026 10:38
@coderabbitai

coderabbitai Bot commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Response snapshots now bound additional outputs to their recorded lengths. Disaggregation transfer-completion and cancellation paths now prepare responses with metadata publication. Tests cover output snapshots, serialization, accumulated log probabilities, and speculative-decoding metadata.

Changes

Response handling

Layer / File(s) Summary
Bound response snapshot outputs
tensorrt_llm/_torch/pyexecutor/llm_request.py, tests/unittest/_torch/pyexecutor/test_request_utils.py
PyResult records additional-output lengths when a snapshot is created. Output access and serialization use those lengths. Tests cover output boundaries, snapshot stability, and accumulated log-probability history.
Publish metadata in disaggregation responses
tensorrt_llm/_torch/disaggregation/orchestration/coordinator.py, tensorrt_llm/_torch/pyexecutor/py_executor.py, tests/unittest/_torch/disaggregation/*
Transfer completion and cancellation prepare responses with metadata publication. Tests check response metadata for fast completion and cancellation, including streaming requests.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Suggested reviewers: juney-nvidia

Merge Risk: Minimal · up to 6ce43

This change makes response snapshots exclude outputs appended later, and makes transfer-completion and cancellation responses carry current draft-token and iteration metadata. No concrete merge-blocking risk was identified, and regression tests cover the changed behavior.

Security architecture risk: Low · up to 6ce43

Most response isolation is preserved, but shared output dictionaries can invalidate an earlier streaming response when routed-expert results are added later. The supported consequence is a narrowly scoped response-read failure, not demonstrated cross-request disclosure or privilege expansion.

Retained concerns

  • Low · reliability · inferred: Shared response snapshots freeze output lengths but not dictionary membership. For a streaming request with nonempty additional outputs and routed-expert capture, completion can add routed_experts to the dictionary retained by earlier snapshots. A delayed in-process additional_generation_outputs read then indexes a missing captured length and raises KeyError, violating the response-lifetime ownership contract. The base detached this dictionary. Serialized responses and snapshots created after route attachment are unaffected; broader failure propagation is not established.
Security review details

Security Blast Radius

  • inferred — The identified failure requires an earlier unconsumed in-process snapshot, a nonempty additional-output dictionary lacking routed_experts, and later route attachment. Evidence supports failure of the affected response read, not cross-request disclosure, privilege gain, or compromise of other services.

Trust Boundaries and Controls

  • observed — Serialization limits additional outputs to the snapshot’s captured key-and-length map. Final response snapshots are created after route attachment in the ordinary finished-request response path, so they capture the added key.

Resilience and Maintainability Implications

  • observed — Existing transfer-drain handling avoids replacing an already committed logical response with a late transfer error. Physical settlement also preserves completed and error states rather than rewriting them during resource release.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main theme of the changeset: improving response handling (published sampler metadata in early-completion and cancellation responses, and snapshot output boundaries) plus added regr…
Description check Passed The description gives a clear summary of what changed and why, with a follow-up reference to #22346. It covers the response metadata fix, the snapshot storage change, the removal of PyResult.Diff, and…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main theme of the changeset: improving response handling (published sampler metadata in early-completion and cancellation responses, and snapshot output boundaries) plus added regression coverage. It follows the required '[TRTLLM-17528][refactor] Summary' format and is specific enough for a teammate scanning history. It does not name every change (for example, removing PyResult.Diff from snapshots), which is acceptable.

Full details: Description check

Explanation

The description gives a clear summary of what changed and why, with a follow-up reference to #22346. It covers the response metadata fix, the snapshot storage change, the removal of PyResult.Diff, and the added regression tests. The template's Description and Test Coverage sections are present but the Test Coverage section is empty, with only the template comment left in place. The PR checklist is checked. The description also carries a copy of the title template comment. The Test Coverage section should list the specific tests that protect the changes, such as the transfer and idle-cancellation tests and the request-utils tests for beam logprobs and serialization bounds. The author should also add a short explanation of the problem under the Description heading, since the current bullets describe the solution only. Because the core content is present, the check passes with these improvements recommended.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 6533070 and 6ce436a.

Files selected for processing: 7

  • tensorrt_llm/_torch/disaggregation/orchestration/coordinator.py
  • tensorrt_llm/_torch/pyexecutor/llm_request.py
  • tensorrt_llm/_torch/pyexecutor/py_executor.py
  • tests/unittest/_torch/disaggregation/coordinator_harness.py
  • tests/unittest/_torch/disaggregation/test_disagg_coordinator_transfers.py
  • tests/unittest/_torch/disaggregation/test_disagg_idle_cancellation.py
  • tests/unittest/_torch/pyexecutor/test_request_utils.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants