Repository navigation
[TRTLLM-17363][feat] Add Beam optimizations and qualified deployment recipes - #21474
Open
Mgluhovskoi wants to merge 81 commits into
Open
Mgluhovskoi wants to merge 81 commits into
Mgluhovskoi wants to merge 81 commits into
Conversation
Mgluhovskoi
marked this pull request as ready for review
November 2, 2026 21:05
Mgluhovskoi
requested review from
2ez4bz,
PerkzZheng,
Shixiaowei02,
SimengLiu-nv,
Tabrizian,
allisonlim-nv,
farazkh80,
jieli-matrix,
kaiyux,
liji-nv,
mikeiovine,
schetlur-nv,
sunnyqgg,
xxi-nv and
yechank-nvidia
November 2, 2026 21:05
Mgluhovskoi
force-pushed
the
mergeback/beam-4-trtllm-17363
branch
from
November 19, 2026 20:54
777aa34 to
496113e
Compare
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
… is safe Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
A deadline can expire on a single stage while another stage still owns destination KV pages. Hold stopped requests out of first-token completion and scheduling until every owner can retire them. Keep independent attention-DP lanes separate and prioritize client aborts over timeouts. Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Managed draft context cannot consume sliding-window scratch pages. Keep target scratch reuse on a separate target manager and clear it in the private draft config. Reject incompatible peer-local choices when disaggregated workers are configured to share. Beam uses the plain V2 manager, so validate pool_ratio against the actual combined target and draft windows before sizing or allocating. Preserve the requested ratio while estimation temporarily clears it. Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add Beam DFlash2 speculative decoding, opt-in FP8 KV cache, native disaggregated serving, and deployment recipes. This is part 4/4 of TRTLLM-17024, covering TRTLLM-17363.
The integration preserves current upstream cache allocation, KV geometry, DFlash context handling and transfer interfaces while retaining Beam scale loading, cache identity and handoff validation. It also carries the reviewed fixes from the preceding PRs.
Beam’s context and generation launch examples both set
TRTLLM_DISAGG_SHARED_DRAFT_KV=1because the DFlash2 handoff requires the drafter’s prompt KV. Other DFlash deployments retain upstream’s worker defaults: with the variable unset, ordinary managers keep draft KV private and eligible specialized managers can share it.Review scope and merge order
#21471 is merged. #21472 and #21473 must land before this PR. All remaining PRs target main, so this PR currently includes its unmerged dependencies.
Incremental changes after #21473.
The review branch originated from the rewritten
NVIDIA:feat/reflection-beamhistory. It now includes main integration and subsequent review fixes; it is no longer an unchanged extraction of that history.Validation
2c5daa292433; native sources and dependency inputs match that build.This PR retains the documented eager execution limits. Dynamo is preview; later CUDA-graph/overlap and Dynamo completion work is separate.
Remaining before merge
2c5daa292433base, including the ModelingV2 package relocation and shared executor/transfer updates; these currently introduce fresh conflicts.No model weights or private evidence/workflow files are included.