Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
035bc6f
[TRTLLM-16186][feat] lend KV cache pages and recurrent state in place
Shixiaowei02 Oct 23, 2026
1a54b26
[TRTLLM-16186][feat] Export the type of a write's arrivals, and tell …
Shixiaowei02 Oct 25, 2026
3b92bef
[TRTLLM-16186][fix] Publish recurrent-state shards over tensor and co…
Shixiaowei02 Oct 25, 2026
e0a56e7
[TRTLLM-16186][refactor] Tell the ranks holding the same bytes beside…
Shixiaowei02 Oct 25, 2026
d7e9842
[TRTLLM-16186][feat] Publish each layer's model layer beside its laye…
Shixiaowei02 Oct 25, 2026
36fd1c3
[TRTLLM-16186][feat] Lent rows name their slots and the regions they …
Shixiaowei02 Oct 25, 2026
4c1f1ca
[TRTLLM-16186][doc] State that admission does not subtract deferred-c…
Shixiaowei02 Oct 25, 2026
e7a7a4a
[TRTLLM-16186][fix] align lender contracts and preserve cache boundaries
Shixiaowei02 Oct 26, 2026
8e9a1c7
[None][refactor] Let KVCacheManagerV2 build the page table its lender…
bo-nv Oct 28, 2026
968ee0f
[None][feat] Describe each buffer as tensors and map two ranks' pages…
bo-nv Oct 28, 2026
ac0463f
[None][refactor] Move native disagg onto the manager's page table and…
bo-nv Oct 28, 2026
0837fde
[None][fix] Pair ranks and elect senders from the same fan
bo-nv Oct 28, 2026
9d23873
[None][refactor] Drop native's per-kind peer policies
bo-nv Oct 28, 2026
1a9642b
[None][refactor] Let managers with several caches per model layer nam…
bo-nv Oct 28, 2026
c1d96a2
[None][feat] Refuse undeclared roles and check declared layouts byte …
bo-nv Oct 28, 2026
6f33ceb
[None][refactor] Ask the cache manager what each request's transfer m…
bo-nv Oct 28, 2026
1b4f650
[None][refactor] Rebuild what a transfer leaves behind through one ma…
bo-nv Oct 28, 2026
f1ff04a
[None][fix] Size an imported request's history from the transfer-leng…
bo-nv Oct 28, 2026
8e7d8ef
[None][refactor] Pair native's pools by where each buffer's bytes go
bo-nv Oct 28, 2026
ee0da85
[None][feat] Give each bounce writer the share it sends, from its own…
bo-nv Oct 28, 2026
861002b
[None][refactor] Transfer by the manager's page table, without a nati…
bo-nv Oct 28, 2026
73be49f
[None][fix] Derive layer group shares from the tensors their buffers …
bo-nv Oct 29, 2026
af820a4
[None][fix] Refuse peers whose element types, windows, sinks or sub-p…
bo-nv Oct 29, 2026
5f32e58
[None][refactor] Name the roles a receiver rebuilds and require their…
bo-nv Oct 29, 2026
d4ec6b4
[None][fix] Move recurrent state from its cache's slot and never in p…
bo-nv Oct 29, 2026
5e7e4d1
[None][fix] Hold per-request transfer hooks to the forms transfers read
bo-nv Oct 29, 2026
9b57123
[None][test] Refuse stray and out-of-slot writes in the page layout o…
bo-nv Oct 29, 2026
e76cb3a
[None][doc] Name the declarations attach_lender refuses
bo-nv Oct 29, 2026
ab28a7a
[None][fix] Name the base manager's retained-precision twins again
bo-nv Oct 29, 2026
6326c03
[None][fix] Leave retention windows out of the peer correspondence check
bo-nv Oct 29, 2026
3cf01f3
[None][fix] Refuse an index-key role the manager does not declare
bo-nv Oct 29, 2026
2e615f3
[None][fix] Name shares by every buffer the lender lends and declare …
bo-nv Oct 29, 2026
1529765
[None][fix] Tie added derived roles to their own rebuild and uneven s…
bo-nv Oct 29, 2026
2566e18
[None][fix] Refuse partial prompt transfers under Helix when the tran…
bo-nv Oct 29, 2026
928c0f4
[None][test] Give the oracle a slot per layer group of a shared pool …
bo-nv Oct 29, 2026
c710e15
[None][fix] Name mixed-precision layers alike on every pipeline stage…
bo-nv Oct 29, 2026
35a55de
[None][fix] Refuse a partial prompt transfer under Helix bound on the…
bo-nv Oct 29, 2026
a25c389
[None][doc] State what the transfer guide's managers declare and peer…
bo-nv Oct 29, 2026
cddd99b
[None][fix] Refuse peers naming a model layer both hold by other layers
bo-nv Oct 29, 2026
5e25330
[None][fix] Refuse layer names a stage cannot give and head splits of…
bo-nv Oct 29, 2026
972ae3c
[None][doc] Say which naming the transfer guide enforces and what par…
bo-nv Oct 29, 2026
cfb4966
[None][fix] Compare the global layers of each cache of a model layer …
bo-nv Oct 29, 2026
b7b8299
[None][fix] Refuse a buffer of several sub-pages split over ranks
bo-nv Oct 29, 2026
b45a7be
[None][doc] Bring the transfer guide's naming, pools and pipelining n…
bo-nv Oct 29, 2026
088052e
[None][test] Expect the base manager to refuse an undeclared index-ke…
bo-nv Oct 29, 2026
5a20379
[None][fix] Refuse a cache a peer does not send for a model layer it …
bo-nv Oct 29, 2026
cfd22cd
[None][fix] Let buffers that never move or move whole keep several su…
bo-nv Oct 29, 2026
feaaa3c
[None][doc] Describe request-rebuilt roles, stage pairing and CSA2's …
bo-nv Oct 29, 2026
c63053f
[None][fix] Refuse rebuilt-per-request roles without the request hook…
bo-nv Oct 30, 2026
8d067aa
[None][fix] Hash model layers into the lender's layout id
bo-nv Oct 30, 2026
1a359e0
[None][fix] Refuse internal layers the default naming would take for …
bo-nv Oct 30, 2026
50313e0
[None][test] Refuse stage-placed caches under other splits of as many…
bo-nv Oct 30, 2026
262ddac
[None][doc] State the request-rebuilt, sub-page and naming rules as t…
bo-nv Oct 30, 2026
5eba128
[None][doc] Describe and pin how the lender's layout document writes …
bo-nv Oct 30, 2026
9237117
[None][refactor] Read the page table and transfer queries off layout …
bo-nv Oct 30, 2026
86b2c44
[None][fix] Let managers deriving from a layout policy take their reg…
bo-nv Oct 30, 2026
893e5ab
[None][refactor] Keep the page table and transfer hooks on the cache …
bo-nv Oct 30, 2026
fa34afb
[None][fix] Count a model layer held through any buffer when refusing…
bo-nv Oct 30, 2026
988bf1e
[None][doc] Describe virtual layer naming as the managers build their…
bo-nv Oct 30, 2026
1c60c97
[None][refactor] Hash each axis of a page layout by its fields for th…
bo-nv Nov 3, 2026
ce6b16b
[None][refactor] Give the native mapper each page's position so a buf…
bo-nv Nov 3, 2026
c16ec9a
[None][refactor] Let a layer group's kind pair a request's slots over…
bo-nv Nov 3, 2026
d0ea8e5
[None][refactor] Let an axis' kind say which indices a rank holds and…
bo-nv Nov 3, 2026
0129eb6
[None][refactor] Derive what moves between ranks from what each axis …
bo-nv Nov 4, 2026
31f4e7f
[None][feat] Map rings of token positions between ranks that size the…
bo-nv Nov 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions docs/source/developer-guide/kv-transfer.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ The implementation described here is the Python transceiver, which lives under [
The control plane — discovery, per-request metadata, and completion notices — is ZeroMQ. The data plane is one-sided NIXL writes issued by the context worker into memory the generation worker has registered. The two never share a channel.

1. Context phase completes computation, KV cache stays in device memory awaiting transmission. The context worker publishes only the endpoint where it can be reached, on `ContextPhaseParams.disagg_info_endpoint`.
2. On first contact with a context instance, the generation worker fetches that instance's `RankInfo` — topology, attention geometry, serialized page table, transfer agent descriptor — from the context rank 0, then registers its own `RankInfo` with every context rank. The result is cached per endpoint, so only the first request pays for the round trip.
2. On first contact with a context instance, the generation worker fetches that instance's `RankInfo` — topology, the manager's page table, transfer agent descriptor — from the context rank 0, and from each context rank that may write its pages that rank's own `RankInfo`, whose page layout says what it sends (which sizes its share of a bounce region, and paired context ranks that do not hold every index of a tensor the generation rank holds are refused here; which pipeline stages pair is decided by their layer counts); it then registers its own `RankInfo` with every context rank. The result is cached per endpoint, so only the first request pays for the round trips.
3. Generation phase requests KV cache for specific tokens, sending its destination block IDs to each overlapping context rank.
4. Context computes both source and destination addresses locally and submits a one-sided NIXL transfer. On the default path that transfer carries one descriptor per cache fragment; only the optional bounce path coalesces a request into a single contiguous buffer and a single write. All address arithmetic happens on the context side; the generation worker's CPU is not involved.
5. Context reports completion over the control plane. The receive session completes once every expected writer has reported.
Expand All @@ -43,11 +43,11 @@ The control plane — discovery, per-request metadata, and completion notices

### Peer Registrar and Mappers

`native/peer.py` resolves which peer ranks hold the same layers when context and generation use different TP/PP/DP configurations, and elects exactly one owner per destination so nothing is transferred twice. `native/mixers/` holds the per-architecture layout policies that turn a matched pool pair into `(src_ptrs, dst_ptrs, sizes)` — separate mappers for matching head counts, for head-major and token-major mismatches, for replicated pools, and for Mamba conv and SSM state.
`native/peer.py` pairs each rank with the peer ranks whose share of every parallel dimension overlaps its own (`layout_mapping.fan`), checks at registration that the two ranks' page layouts correspond, and groups the runs a rank sends a peer rank by the pool each run leaves and the peer's pool it lands in. Which bytes a rank sends a peer rank follows from the two ranks' page layouts alone (`kv_cache/layout_mapping.py`, `map_buffers`): every buffer is the local pieces of tensors whose axes say how they split over tensor and context ranks, and of the indices both ranks hold the source sends those it is elected for among the paired ranks holding them, so nothing is sent twice and no cache kind or model is special-cased. `native/runs.py`'s `RunsMapper` adds the resulting byte runs to each block's or slot's base pointers. The runs are computed once per peer rank and are the same for every page, except those of an axis of token positions (an axis whose kind is positional, as `Ring`, FP4 MLA's high-precision tail, is), whose place for a position depends on each side: for those, `layout_mapping.page_runs` moves every position of a page both sides hold (for a ring, the latest positions both rings keep), each from its index on the source to its index on the destination as the kind places it, within the request's transfer window (`TransferWindow`), which the transfer passes for every page. Which slots of a request pair in one transfer is the layer group's kind's to say (`layout_mapping.pair_pages`: a slot per page for attention, Helix included, one per request for recurrent state).

### Page Table

`resource/page.py` describes a rank's KV storage independently of which cache manager produced it, which is what lets a V1 context worker talk to a V2 generation worker. `resource/kv_extractor.py` builds it from either manager generation and turns block IDs into slot pointers; `resource/cache_reuse.py` reports the cached prefix length uniformly, so context-side reuse, generation-side reuse, and sliding-window stale prefixes all resolve without a special case.
`KVCacheManagerV2.get_page_table()` describes how the manager lays out its pages (`kv_cache/page_layout.py`): each buffer's tensors, where it lies in its pool slot, and where the rank stands. Native uses that table as it is: it travels to peers in `RankInfo` (`page_table_to_dict`), and `resource/utils.py` turns block IDs into slot pointers from its device pools; `resource/cache_reuse.py` reports the cached prefix length uniformly, so context-side reuse, generation-side reuse, and sliding-window stale prefixes all resolve without a special case.

### Transfer Agent

Expand All @@ -63,13 +63,15 @@ Note also that `cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/` outliv

### Supporting a New Cache Layout

A new storage layout usually means a new mapper under `native/mixers/` plus a page-table builder change in `resource/kv_extractor.py`. Both peers must agree: pool role and mapper kind are matched exactly, and a mismatch raises rather than silently transferring a subset.
A new cache's manager declares how its buffers spread over ranks by overriding a few `KVCacheManagerV2` hooks: `_page_buffer_tensors` (built from `page_layout`'s `head_major`, `token_major`, `sectioned`, `replicated` and `ring`, or declaring axes of its own; an axis whose indices spread in a way none of the axis kinds states takes a new kind in `page_layout` saying which indices each rank holds, or which token positions a page holds and at which index, from which `layout_mapping` derives what moves (only a kind whose ranks hold overlapping indices that are not runs registers `owned_indices`), and a layer group addressing a request's pages otherwise a new layer-group kind, which `_page_group_kind` returns and which registers `pair_slots`, leaving the transfer unchanged), `_page_untransferred_roles` for buffers that never move because they hold nothing a peer needs and `_page_derived_roles` for those the receiver rebuilds from the transferred ones (a manager naming any overrides `rebuild_disagg_imported_cache`, or its page table is refused), `_page_request_rebuilt_roles` for those it rebuilds, receiving, for some requests instead (so a peer may hold none of them for a model layer it holds; a manager naming any overrides `disagg_untransferred_buffers`, or its page table is refused), and, when its layers or heads differ from the base manager's, `_page_layer_ids` and `_page_kv_heads`; recurrent state splits over the ranks `_page_state_split` names, as its pools are sized. The base manager declares only K/V and their block scales, split by head and head-major (HND) as the TRTLLM attention kernels write them (a manager whose kernels write them token-major, NHD, declares `token_major`); every other role, `Role.INDEX_KEY` included, is declared by the manager registering it, and a role a manager adds without declaring it, or tensors whose pieces on a rank do not add up to a buffer's size, are refused when the page table is built, rather than guessed. Peers pair buffers by global layer id, so each cache of a model layer goes by the same global ids on every pipeline stage of every peer, and peers naming a cache both transfer by other ids are refused: `_page_layer_ids` derives it from the model layer alone, or from the model layer and attention type for a manager keeping several caches per model layer (`_virtual_layers`, from `_layer_attn_to_layer_id`, which such a manager sets on every stage), never from what a stage happens to hold; likewise the model layer a cache is attached to must not depend on which layers a stage holds. The default `_page_layer_ids` names internal layer `i` by the stage's `i`-th model layer and refuses a manager whose `layer_offsets` numbers its layers otherwise. The byte oracle cannot see a misnaming every side shares, so a new cache's read-back test should pair context and generation workers of different pipeline splits. Each layer group's shard, which names its blocks to the lender, follows from the indices its buffers' tensors hold, so ranks holding the same bytes share names. What a transfer of one request moves is the manager's to say as well, through its `disagg_*` methods (a transfer leaving part of the prompt, a buffer or the first tokens behind needs the receiver's executor to recompute them, which today only CSA2's remote-tail replay does): how much of the prompt (`disagg_transfer_length`; a manager moving part of it is refused under Helix), which buffers the receiver rebuilds instead (`disagg_untransferred_buffers`), whether the first tokens travel (`disagg_sends_first_tokens`), which slot holds each state layer group's recurrent state (`disagg_state_slot`, by default the one the request's cache holds; a state group without one raises rather than moving nothing), and whether the pages can move in pieces while the prompt is computed (`disagg_pipelined_transfer_blocker`, by default not where the model keeps recurrent state, `model_has_recurrent_state`); caches the generation worker derives from imported ones are rebuilt in `rebuild_disagg_imported_cache` before a request's first forward. The transfer itself needs no change. It moves only what lies in the manager's V2 pools: request state the generation worker reads from anywhere else never travels, so a manager keeping such state refuses disaggregated serving, as MiniMax M3 does for its plain-tensor index-V. Peers whose layouts do not correspond — a buffer one side transfers that the other lacks, a cache of a model layer named by other global layers or a global layer naming another model layer, tensors of different extents or element types (a tensor states its type with `dtype`; a ring of token positions may hold different numbers of slots on the two sides, a worker sizing it by its speculative rewind), KV of another element type or quantization as the managers report them (`_page_dtype`, `_page_quant_mode`; a mixed-precision manager reports its high-precision type), other sub-pages (a buffer of several sub-pages per block that moves is declarable only replicated or as each rank's own share) or a layer group of another kind, or paired source ranks that do not hold every index of a tensor a destination holds — are refused at registration rather than silently transferring a subset or bytes of another meaning. A role of a model layer both sides hold that one side alone transfers is refused, whichever side sends, unless the side holding it rebuilds it per request (`_page_request_rebuilt_roles`), as CSA2's generation workers under bounded replay rebuild the sliding caches a context-only worker keeps none of; such a role, on a global layer of its own or on one the other side holds too, is registered on the destination and left out of a request's transfer through `disagg_untransferred_buffers`. A model layer no source holds is not refused by the layouts. How many tokens each side keeps of a windowed layer is not compared: retention margins such as reuse backoff and draft tokens legitimately differ between the sides, and a transfer moves the blocks both sides still hold.

Unit tests for these seams live in `tests/unittest/disaggregated/` — byte-level transfer checks, topology and mapper selection, page-table construction, and multi-process end-to-end coverage.
To test a new cache, add a `tests/unittest/disaggregated/test_<model>_kv_transfer.py` on `kv_transfer_harness.run_kv_transfer_test` with a manager factory and hooks that write and read the cache through the model's own tensor views, over the context/generation topologies the model is served with. The harness first checks the layouts the managers declare against each other byte for byte (`tests/unittest/utils/page_layout_oracle.py`, usable on its own for CPU checks of hand-written or manager-built layouts), then runs a real transfer and hands the received caches to the read-back hook.

Unit tests for these seams live in `tests/unittest/disaggregated/` — byte-level transfer checks, rank pairing, page-table construction, and multi-process end-to-end coverage; `tests/unittest/_torch/executor/kv_cache/test_layout_mapping.py` checks the mapping byte by byte across TP, attention DP, Helix, PP, uneven head splits and rings of token positions sized differently on the two sides.

## Current Limitations

- **NIXL only.** MPI, UCX, and Mooncake deployments stay on the C++ transceiver.
- **One KV slice per request.** The whole prompt moves in a single transfer after prefill completes. The session machinery is multi-slice capable and `KVSlice.layer_range` exists, but no producer sets it — there is no layer-wise or chunked transfer overlapping compute yet.
- **Chunked transfer only through pipelined transfer.** By default the whole prompt moves in a single transfer after prefill completes. `cache_transceiver_config.enable_pipelined_transfer` sends each completed prefill chunk while later chunks compute; the transceiver refuses it at start with the Python bounce buffer, context parallelism, block reuse other than `all_reusable`, or a manager whose `disagg_pipelined_transfer_blocker` names a reason (by default a model keeping recurrent state); the executor refuses CSA2 remote-tail requests one by one. `KVSlice.layer_range` exists but no producer sets it, so there is no layer-wise transfer.
- **Both sides must run the same build and the same transceiver runtime.** `RankInfo` is serialized field for field and carries no version tag, so there is nothing to negotiate against. Pairing a C++ context worker with a Python generation worker fails on the first request, because the context worker publishes no `disagg_info_endpoint`.
- **Submitted NIXL operations cannot be aborted.** Cancellation is cooperative: an operation already handed to the transfer agent runs to completion. Lifecycle ownership is not complete either — safe retirement of a session after an ambiguous failure or a cancellation is not fully enforced, so do not assume the KV pages are reclaimed at a well-defined point on those paths.
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

from __future__ import annotations

from typing import Any, Optional, Tuple
from typing import TYPE_CHECKING, Any, Optional, Tuple

from tensorrt_llm.llmapi.llm_args import MTPDecodingConfig
from tensorrt_llm.quantization.mode import QuantAlgo
Expand All @@ -22,18 +22,25 @@
# fine-tunes of the same shape; the accuracy gate pins identity.
_SHAPE = (78, 6144, 256, 2048)

if TYPE_CHECKING:
from tensorrt_llm._torch.model_config import ModelConfig
from tensorrt_llm.llmapi.llm_args import TorchLlmArgs


def route(
pretrained_config: Any,
mapping: Any,
compute_capability: Tuple[int, int],
spec_config: Optional[Any],
kv_cache_quant_algo: Optional[Any],
model_config: Optional[ModelConfig] = None,
llm_args: Optional[TorchLlmArgs] = None,
) -> type:
"""Return the target class for this deployment, or raise ``NoTarget`` saying why none fits.

Each target is imported where it is returned, so declining costs no import
of modeling code.
of modeling code. No engine choice in ``model_config`` or ``llm_args`` changes
which target runs, so neither is read.
"""
shape = (
pretrained_config.num_hidden_layers,
Expand Down
Loading