Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,7 @@ Encoder and LLM forward Params are execution views.
| **Attention backends** | `TorchLlmArgs.attn_backend` selects kernel: `TRTLLM` (default), `FlashInfer`, `FlashAttention` |
| **Distributed execution** | Tensor/pipeline parallelism via `Mapping` class, multiple backends (MPI, Ray, RPC) |
| **Auto-discovery** | Models self-register via `automodel.py`, resolved by HF config `architectures` field |
| **Per-step device copies** | New per-step H2D/D2D copies in runtime code go through `_torch/pyexecutor/device_copy.py` with a named `CopySite`; a model opts in by declaring a `CopyPlan` (`copy_plan`). The step-input pack (`step_inputs.py`, `h2d_cache.py`, `tensorrt_llm/_device_copy.py`, `RuntimeHooks.pack_*` / `skip_unchanged_h2d`) is legacy: keep its users working, add no new ones. See [the copy plan guide](docs/source/developer-guide/device-copy-plan.md) |

## VisualGen

Expand All @@ -158,6 +159,7 @@ Key files:
- **Copyright year** — update to current year when modifying existing files; add full header to new files.
- **Avoid broad exception handling** — catch specific exceptions, not bare `except:` (see `CODING_GUIDELINES.md`).
- **One concern per PR** — avoid scope creep. If a PR touches unrelated areas, split it.
- **Replacing a shared runtime mechanism** — add the new one beside the old, opt-in and off by default, so code on the old one keeps its behavior and performance; move models in follow-up PRs and delete the old mechanism last. Do not delete or change the old API in the PR that introduces the new one.
- **User-facing configuration classes** - when editing or defining any user-facing configuration classes (particularly `BaseLlmArgs` or any class used in its fields), you **MUST** follow the Pydantic guidelines in `CODING_GUIDELINES.md`.

## Development Workflow
Expand Down
75 changes: 75 additions & 0 deletions docs/source/developer-guide/device-copy-plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
(device-copy-plan)=

# Per-Step Device Copies: the Copy Plan

Between two decode CUDA graph replays, the next step's inputs and metadata reach their graph-stable device buffers as
a chain of small copies (position ids, KV lengths, page tables, the graph runner's static inputs, speculative RNG
seeds). Each one is a separate operation on the stream that the next graph waits for, so the GPU idles for the whole
chain. The copy plan in `tensorrt_llm/_torch/pyexecutor/device_copy.py` is the runtime's mechanism for these copies.
New runtime code that issues per-step copies uses it.

## The mechanism

Two questions decide what happens to a per-step copy, and two different owners answer them.

* **When the copy must land** is decided by the model, because it depends on who reads the destination. Every copy in
shared runtime code names its call site with a `CopySite` and goes through `device_copy.copy` (or
`device_copy.fill_tail`). A model declares a `CopyPlan` as `copy_plan` on the model:
* `merge`: the sites whose copies are staged into the step's pack. The pack is issued as one upload and one scatter
launch, at the latest when the graph replays.
* `flush_after`: barrier sites, whose staged copy must land before anything that runs next.
* `parallel_scatter`, `changed_span` and `final_flush=InGraph(...)`: how the pack is issued. `InGraph` makes the
replayed graph's own prologue issue it.
* **Whether the copy is needed at all** is decided by the buffer owner, because it depends on the values and on who
else writes the buffer. The owner passes `unchanged=True` when the destination already holds the values, or keeps a
`device_copy.DeviceCopyBuffer` with the copy's site as its record of what the buffer holds.

`TLLM_DEVICE_COPY_CHECK=1` verifies at runtime what a plan and its owners assert:
* that nothing writes a merged destination before its flush;
* that every skipped (`unchanged`) copy really holds its values.

The unit tests run with this check on (`tests/unittest/conftest.py`).

## Adopting the copy plan in a model

1. Name the model's own copy sites in a `CopySiteEnum` of its own, next to the code that issues them. Shared sites are
in `device_copy.CopySite`.
2. Build the plan from the readers of each destination:
* merge what no device work touches before the graph replay;
* anchor a `flush_after` barrier on each copy that an eager reader consumes;
* set the plan as the model's `copy_plan`.
3. Let each buffer owner skip its unchanged uploads on its own, with a `DeviceCopyBuffer` that has the copy's site.
4. Turn off the model's `RuntimeHooks` step-input pack switches (`pack_step_inputs`, `skip_unchanged_h2d` and the
`pack_*` options). The engine refuses a model that declares a plan and also turns these on (`check_one_copy_path`).
5. Run the model's tests with `TLLM_DEVICE_COPY_CHECK=1`, and compare its step time with the step-input pack it
replaces.

## Coexistence with the step-input pack

The copy plan replaces an older mechanism:
* the step-input pack, `pyexecutor/step_inputs.py`;
* the skip cache, `pyexecutor/h2d_cache.py`;
* `tensorrt_llm/_device_copy.DeviceCopyBuffer`;
* the `RuntimeHooks` switches that turn them on.

Both mechanisms are supported while models move over:

* A model that declares no copy plan has no copier, and its copies run exactly as before. A shared call site that has
a `CopySite` takes its original path whenever no copier is active (`device_copy.current()` is None).
* A model that declares a plan has a copier. The shared sites route through it, and the step-input pack stays off.
* A model uses one mechanism for all of its step's copies, never both.

The rules while both exist:

* New per-step copies in runtime code go through `device_copy` with a named site.
* Do not add new users of `step_inputs`, `h2d_cache`, `tensorrt_llm/_device_copy` or the `RuntimeHooks` step-input
pack switches. Existing users keep working until their model moves to a copy plan.
* When a shared call site gains a `CopySite`, keep its original path for engines without a copier, so that models
without a plan see no behavior or performance change.
* Once every model has moved, a follow-up change removes the step-input pack, the skip cache and their `RuntimeHooks`
switches.

The same approach applies to any shared runtime mechanism that a new one replaces:
1. Add the new mechanism beside the old one, opt in and off by default.
2. Move the models in follow-up changes.
3. Remove the old mechanism last.
1 change: 1 addition & 0 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ Welcome to TensorRT LLM's Documentation!
developer-guide/kv-cache-cold-page-codec.md
developer-guide/kv-cache-compression-development.md
developer-guide/sparse-attention-development-guide.md
developer-guide/device-copy-plan.md
developer-guide/telemetry.md


Expand Down
24 changes: 20 additions & 4 deletions tensorrt_llm/_torch/attention/backends/trtllm.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,9 @@
from ...distributed import cp_allgather
from ...kimi_k3_cache_policy import get_kimi_k3_fp8_kv_layer_ids, is_kimi_k3
from ...locality_domain.runtime import LocalityDomainRuntime
from ...pyexecutor import h2d_cache, step_inputs
from ...pyexecutor import device_copy, h2d_cache, step_inputs
from ...pyexecutor.config_utils import is_mla
from ...pyexecutor.device_copy import CopySite
from ...pyexecutor.h2d_cache import copy_h2d_pool_shared
from ...utils import (compute_swizzled_sf_shape, get_global_attrs,
get_model_extra_attrs, helix_local_len_tensor)
Expand Down Expand Up @@ -1366,7 +1367,14 @@ def prepare(self) -> None:
device='cpu',
)
self.prompt_lens_cpu[:self.num_seqs].copy_(prompt_lens)
if not self._upload_to_device(
if device_copy.current() is not None:
# The model declares a copy plan, which owns the engine's copies. Never
# skipped: CUDA graph metadata of every graph key takes this buffer from
# one pool, so no host record proves its device contents.
device_copy.copy(CopySite.ATTN_PROMPT_LENS,
self.prompt_lens_cuda[:self.num_seqs],
self.prompt_lens_cpu[:self.num_seqs])
elif not self._upload_to_device(
"prompt_lens", self.prompt_lens_cuda[:self.num_seqs],
self.prompt_lens_cpu[:self.num_seqs], h2d_cache.enabled()):
copy_h2d_pool_shared(self.prompt_lens_cuda[:self.num_seqs],
Expand Down Expand Up @@ -1426,7 +1434,11 @@ def prepare(self) -> None:
# the sequence length including the cached tokens and the input tokens.
self.kv_lens[:self.num_seqs].copy_(
kv_lens + self.kv_cache_params.num_extra_kv_tokens)
if not self._upload_to_device(
if device_copy.current() is not None:
device_copy.copy(CopySite.ATTN_KV_LENS,
self.kv_lens_cuda[:self.num_seqs],
maybe_pin_memory(kv_lens[:self.num_seqs]))
elif not self._upload_to_device(
"kv_lens", self.kv_lens_cuda[:self.num_seqs],
kv_lens[:self.num_seqs], step_inputs.is_open()):
step_inputs.copy_(self.kv_lens_cuda[:self.num_seqs],
Expand Down Expand Up @@ -1777,7 +1789,11 @@ def prepare_encoder_decoder_from_precomputed_lengths(
f"({self.kv_cache_manager.max_seq_len}).")

num_seqs = self.num_seqs
copy_h2d_pool_shared(self.prompt_lens_cuda[:num_seqs], prompt_lens)
if device_copy.current() is not None:
device_copy.copy(CopySite.ATTN_PROMPT_LENS,
self.prompt_lens_cuda[:num_seqs], prompt_lens)
else:
copy_h2d_pool_shared(self.prompt_lens_cuda[:num_seqs], prompt_lens)
self.kv_lens_cuda[:num_seqs].copy_(kv_lens, non_blocking=True)
self.host_total_kv_lens[0] = context_kv_tokens
self.host_total_kv_lens[1] = generation_kv_tokens
Expand Down
40 changes: 36 additions & 4 deletions tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@
from ..speculative.spec_sampler_base import SampleStateTensorsSpec
from ..speculative.utils import get_draft_kv_cache_manager
from ..utils import decode_stream_priority, make_weak_ref, piecewise_cuda_graph
from . import step_inputs
from . import device_copy, step_inputs
from .device_copy import CopySite
from .llm_request import LlmRequest, get_draft_token_length
from .replay_inputs import stage_replay_inputs
from .resource_manager import (BaseResourceManager, NoFreeSlotsError,
Expand Down Expand Up @@ -363,6 +364,9 @@ def __init__(self, config: CUDAGraphRunnerConfig):
self.graphs: Dict[KeyType, torch.cuda.CUDAGraph] = {}
# Graphs captured with a step-input pack's prologue, and that pack (step_inputs.capture_prologue).
self._step_input_prologue_keys: Dict[KeyType, Any] = {}
# Graphs captured with a copier's in-graph final flush prologue, and that copier
# (device_copy.capture_prologue).
self._prologue_copiers: Dict[KeyType, Any] = {}
self.graph_outputs: Dict[KeyType,
Callable[[], Optional[torch.Tensor]]] = {}
self.graph_metadata: Dict[KeyType, Dict[str, Any]] = {}
Expand Down Expand Up @@ -1429,6 +1433,10 @@ def run_warmup():
step_inputs.capture_prologue())
if prologue_pack is not None:
self._step_input_prologue_keys[key] = prologue_pack
prologue_copier = (None if self._pre_replay_hooks else
device_copy.capture_prologue())
if prologue_copier is not None:
self._prologue_copiers[key] = prologue_copier
output = forward_fn(capture_inputs)
if len(HOSTFUNC_USER_DATA_HANDLES) > num_hostfuncs:
self._hostfunc_graph_keys.add(key)
Expand All @@ -1443,6 +1451,16 @@ def run_warmup():
self.memory_pool = graph.pool()
return graph_output

@staticmethod
def _copy_static_input(site: CopySite, dst: torch.Tensor,
src: torch.Tensor) -> None:
"""Copy a step input into a graph's static input: through ``site`` when the model
declares a copy plan (an active copier), else through the step-input pack."""
if device_copy.current() is not None:
device_copy.copy(site, dst, src)
else:
step_inputs.copy_(dst, src)

def replay(
self,
key: KeyType,
Expand Down Expand Up @@ -1498,17 +1516,20 @@ def replay(
static_tensors["position_ids"])
hook_positions = self._pre_replay_position_ids[..., :seqlen]
# One launch copies the ids and positions and writes the hooks' positions,
# unless an open step-input pack still defers copies into its sources.
# unless an open step-input pack or copy-plan pack still defers copies into
# their sources.
staged = (self.config.stage_decode_inputs and copy_input_ids
and not self.config.use_mrope and not step_inputs.is_open()
and not device_copy.is_open()
and tuple(position_ids.shape) == (1, seqlen)
and stage_replay_inputs(
static_input_ids, input_ids,
static_tensors["position_ids"][:, :seqlen], position_ids,
hook_positions, None if hook_positions is None else
position_id_offsets[:seqlen]))
if copy_input_ids and not staged:
step_inputs.copy_(static_input_ids, input_ids)
self._copy_static_input(CopySite.GRAPH_INPUT_IDS, static_input_ids,
input_ids)

if self.config.use_mrope:
expected_position_ids_shape = (3, 1, seqlen)
Expand Down Expand Up @@ -1559,7 +1580,8 @@ def replay(
static_position_ids = static_tensors["position_ids"][:, :seqlen]
if (not prestaged and not staged and position_ids.data_ptr()
!= static_position_ids.data_ptr()):
step_inputs.copy_(static_position_ids, position_ids)
self._copy_static_input(CopySite.GRAPH_POSITION_IDS,
static_position_ids, position_ids)

num_encoder_tokens = key.num_encoder_tokens
if num_encoder_tokens:
Expand Down Expand Up @@ -1593,6 +1615,13 @@ def replay(
if prologue_pack is not None and not captured_pack_active:
# Replayed outside the pack's context: nothing filled its window.
prologue_pack.quiesce_graph_window()
# The same for the copy plan's merged copies (a model that declares a plan).
prologue_copier = self._prologue_copiers.get(key)
captured_copier_active = (prologue_copier is not None
and device_copy.is_active(prologue_copier))
device_copy.flush(in_graph=captured_copier_active)
if prologue_copier is not None and not captured_copier_active:
prologue_copier.quiesce_graph_window()
if self._pre_replay_hooks:
hook_tensors = static_tensors
if hook_positions is not None:
Expand All @@ -1609,6 +1638,8 @@ def replay(
finally:
if prologue_pack is not None:
prologue_pack.on_graph_replay()
if prologue_copier is not None:
prologue_copier.on_graph_replay()
if key in self._hostfunc_graph_keys:
record_hostfunc_graph_replay()
output_ref = self.graph_outputs[key]
Expand Down Expand Up @@ -2219,6 +2250,7 @@ def clear(self, *, release_nccl_window_owners: bool = True):
self._hostfunc_graph_keys.clear()
self._pre_replay_position_ids = None
self._step_input_prologue_keys.clear()
self._prologue_copiers.clear()
self.reset_padding_dummies()
del self.memory_pool
self.memory_pool = None
Expand Down
Loading
Loading