Skip to content

[TRTLLM-17363][feat] Add Beam optimizations and qualified deployment recipes - #21474

Open
Mgluhovskoi wants to merge 81 commits into
NVIDIA:mainfrom
Mgluhovskoi:mergeback/beam-4-trtllm-17363
Open

Mgluhovskoi wants to merge 81 commits into
NVIDIA:mainfrom
Mgluhovskoi:mergeback/beam-4-trtllm-17363

Conversation

@Mgluhovskoi

@Mgluhovskoi Mgluhovskoi commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Add Beam DFlash2 speculative decoding, opt-in FP8 KV cache, native disaggregated serving, and deployment recipes. This is part 4/4 of TRTLLM-17024, covering TRTLLM-17363.

The integration preserves current upstream cache allocation, KV geometry, DFlash context handling and transfer interfaces while retaining Beam scale loading, cache identity and handoff validation. It also carries the reviewed fixes from the preceding PRs.

Beam’s context and generation launch examples both set TRTLLM_DISAGG_SHARED_DRAFT_KV=1 because the DFlash2 handoff requires the drafter’s prompt KV. Other DFlash deployments retain upstream’s worker defaults: with the variable unset, ordinary managers keep draft KV private and eligible specialized managers can share it.

Review scope and merge order

#21471 is merged. #21472 and #21473 must land before this PR. All remaining PRs target main, so this PR currently includes its unmerged dependencies.

Incremental changes after #21473.

The review branch originated from the rewritten NVIDIA:feat/reflection-beam history. It now includes main integration and subsequent review fixes; it is no longer an unchanged extraction of that history.

Validation

  • Integrated and built against main 2c5daa292433; native sources and dependency inputs match that build.
  • Scoped native checks cover Beam attention, DFlash BF16/FP8, draft-pool provisioning, cache handoff, peer/layout behavior and MPI forwarding. The initial failures were followed by targeted passing reruns after fixture corrections; this is not a single clean rerun of every test on the final head.
  • 81 tracked CI-selection/CBTS checks and scoped pre-commit passed. Local Pyrefly was unavailable.
  • No fresh full-model accuracy run was performed for this integration. Historical first-100 scores in the guide are not current-head qualification.
  • Full GSM8K stays excluded from CI. Large component and distributed qualification remain outside L0.

This PR retains the documented eager execution limits. Dynamo is preview; later CUDA-graph/overlap and Dynamo completion work is separate.

Remaining before merge

  • Integrate newer main changes after the validated 2c5daa292433 base, including the ModelingV2 package relocation and shared executor/transfer updates; these currently introduce fresh conflicts.
  • Resolve remaining review findings and obtain required code-owner approvals.
  • Pass current-head CI and any outstanding native/distributed qualification.
  • Land the dependency PRs in order.

No model weights or private evidence/workflow files are included.

@Mgluhovskoi
Mgluhovskoi force-pushed the mergeback/beam-4-trtllm-17363 branch from 777aa34 to 496113e Compare November 19, 2026 20:54
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
… is safe

Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
A deadline can expire on a single stage while another stage still owns destination KV pages. Hold stopped requests out of first-token completion and scheduling until every owner can retire them. Keep independent attention-DP lanes separate and prioritize client aborts over timeouts.

Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Managed draft context cannot consume sliding-window scratch pages. Keep target scratch reuse on a separate target manager and clear it in the private draft config. Reject incompatible peer-local choices when disaggregated workers are configured to share.

Beam uses the plain V2 manager, so validate pool_ratio against the actual combined target and draft windows before sizing or allocating. Preserve the requested ratio while estimation temporarily clears it.

Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.