Repository navigation
Conversation
Publish current Python sampler progress before an early context-transfer completion creates its native result. Prepare between-batch cancellation responses through the same helper so their counters are published and their speculative metadata is snapshotted instead of reverting to empty defaults. Preserve the native transfer state during response creation and the existing connector-save ordering. Add native serialized-result regressions for decoding iteration at early completion and current cancellation counters alongside an unchanged earlier response. Validation: the new regressions reproduced stale iteration and draft state before the fix. All 186 transfer/idle-cancellation tests and the broader 2241-test CPU suite pass, with 114 skipped and 10 deselected. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Keep request-only PyResult.Diff out of responses. Record per-output list lengths when snapshotting instead of copying all accumulated additional-output lists at every decoding step. Bound both property reads and serialized state to the recorded prefixes so later context or generation chunks cannot leak into queued responses. Cover shared list ownership, independent output boundaries, initially empty outputs, and delayed pickle/cloudpickle round trips. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Model the sampler second step with append_log_probs instead of replacing the history. Assert that beam histories grow and cumulative logprobs advance on the request while the first response retains only its original token logprobs and cumulative totals. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Create a streaming response after two distinct logits chunks with exclusion enabled, using prepare_response without the latest-logits override. Append a third chunk and disable exclusion on the live request, then verify that the retained response still exposes the first chunk while the request exposes the third. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
|
/bot run |
|
PR_Github #86334 [ run ] triggered by Bot. Commit: |
|
PR_Github #86334 [ run ] completed with state
|
|
/bot run |
|
PR_Github #86389 [ run ] triggered by Bot. Commit: |
|
PR_Github #86389 [ run ] completed with state
|
|
No actionable comments were generated in the recent review. WalkthroughResponse snapshots now bound additional outputs to their recorded lengths. Disaggregation transfer-completion and cancellation paths now prepare responses with metadata publication. Tests cover output snapshots, serialization, accumulated log probabilities, and speculative-decoding metadata. ChangesResponse handling
Estimated code review effort: 3 (Moderate) | ~20 minutes Change: Bug fix Suggested reviewers: Merge Risk:
|
Description
Follow-up to #22346.
PyResult.Difffrom response snapshots.Dev Engineer Review
llm_request.py, +39/-6): Snapshots now share output lists and store per-output length bounds. Reads and pickling must truncate to those bounds. A missed bound path would expose later appends. Verify every output accessor and the pickle path.cached_tokenscopy (py_executor.py):prepare_responsereplaces the explicit copy. Confirm thatcached_tokensstill reaches the terminal response.Diff. Confirm no consumer readsDifffrom a snapshot.#71533failed. Pipeline#71587completed as UNSTABLE. The comments give no failing-test details. Current-head CI status is not available.QA Engineer Review
tests/unittest/_torch/disaggregation/coordinator_harness.py,test_disagg_coordinator_transfers.py,test_disagg_idle_cancellation.py, andtests/unittest/_torch/pyexecutor/test_request_utils.py.test-db/orqa/list files changed. Whether these tests are listed in CI is not available.Per-File QA Perspective
coordinator.py:release_transferpublishes metadata for settled active requests. Verify responses after transfer completion include the current metadata.llm_request.py: Output lists are bounded by recorded length. Verify serialized responses exclude outputs added after response creation.py_executor.py: Cancelled disaggregated requests now useprepare_response(publish_metadata=True). Verify cancellation responses and deferred termination.coordinator_harness.py:py_draft_tokensdefaults to an empty list. No QA impact beyond test setup.test_disagg_coordinator_transfers.py: Covers fast-completion metadata. Not verified in CI test lists.test_disagg_idle_cancellation.py: Adds a streaming option and a between-batch cancellation regression. Not verified in CI test lists.test_request_utils.py: Covers beam logprob history and response snapshot bounds. Not verified in CI test lists.Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.