diff --git a/AGENTS.md b/AGENTS.md index 9fbcacf0e388..c6f65cec60c0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -133,6 +133,7 @@ Encoder and LLM forward Params are execution views. | **Attention backends** | `TorchLlmArgs.attn_backend` selects kernel: `TRTLLM` (default), `FlashInfer`, `FlashAttention` | | **Distributed execution** | Tensor/pipeline parallelism via `Mapping` class, multiple backends (MPI, Ray, RPC) | | **Auto-discovery** | Models self-register via `automodel.py`, resolved by HF config `architectures` field | +| **Per-step device copies** | New per-step H2D/D2D copies in runtime code go through `_torch/pyexecutor/device_copy.py` with a named `CopySite`; a model opts in by declaring a `CopyPlan` (`copy_plan`). The step-input pack (`step_inputs.py`, `h2d_cache.py`, `tensorrt_llm/_device_copy.py`, `RuntimeHooks.pack_*` / `skip_unchanged_h2d`) is legacy: keep its users working, add no new ones. See [the copy plan guide](docs/source/developer-guide/device-copy-plan.md) | ## VisualGen @@ -158,6 +159,7 @@ Key files: - **Copyright year** — update to current year when modifying existing files; add full header to new files. - **Avoid broad exception handling** — catch specific exceptions, not bare `except:` (see `CODING_GUIDELINES.md`). - **One concern per PR** — avoid scope creep. If a PR touches unrelated areas, split it. +- **Replacing a shared runtime mechanism** — add the new one beside the old, opt-in and off by default, so code on the old one keeps its behavior and performance; move models in follow-up PRs and delete the old mechanism last. Do not delete or change the old API in the PR that introduces the new one. - **User-facing configuration classes** - when editing or defining any user-facing configuration classes (particularly `BaseLlmArgs` or any class used in its fields), you **MUST** follow the Pydantic guidelines in `CODING_GUIDELINES.md`. ## Development Workflow diff --git a/docs/source/developer-guide/device-copy-plan.md b/docs/source/developer-guide/device-copy-plan.md new file mode 100644 index 000000000000..e86ceb0b4783 --- /dev/null +++ b/docs/source/developer-guide/device-copy-plan.md @@ -0,0 +1,75 @@ +(device-copy-plan)= + +# Per-Step Device Copies: the Copy Plan + +Between two decode CUDA graph replays, the next step's inputs and metadata reach their graph-stable device buffers as +a chain of small copies (position ids, KV lengths, page tables, the graph runner's static inputs, speculative RNG +seeds). Each one is a separate operation on the stream that the next graph waits for, so the GPU idles for the whole +chain. The copy plan in `tensorrt_llm/_torch/pyexecutor/device_copy.py` is the runtime's mechanism for these copies. +New runtime code that issues per-step copies uses it. + +## The mechanism + +Two questions decide what happens to a per-step copy, and two different owners answer them. + +* **When the copy must land** is decided by the model, because it depends on who reads the destination. Every copy in + shared runtime code names its call site with a `CopySite` and goes through `device_copy.copy` (or + `device_copy.fill_tail`). A model declares a `CopyPlan` as `copy_plan` on the model: + * `merge`: the sites whose copies are staged into the step's pack. The pack is issued as one upload and one scatter + launch, at the latest when the graph replays. + * `flush_after`: barrier sites, whose staged copy must land before anything that runs next. + * `parallel_scatter`, `changed_span` and `final_flush=InGraph(...)`: how the pack is issued. `InGraph` makes the + replayed graph's own prologue issue it. +* **Whether the copy is needed at all** is decided by the buffer owner, because it depends on the values and on who + else writes the buffer. The owner passes `unchanged=True` when the destination already holds the values, or keeps a + `device_copy.DeviceCopyBuffer` with the copy's site as its record of what the buffer holds. + +`TLLM_DEVICE_COPY_CHECK=1` verifies at runtime what a plan and its owners assert: +* that nothing writes a merged destination before its flush; +* that every skipped (`unchanged`) copy really holds its values. + +The unit tests run with this check on (`tests/unittest/conftest.py`). + +## Adopting the copy plan in a model + +1. Name the model's own copy sites in a `CopySiteEnum` of its own, next to the code that issues them. Shared sites are + in `device_copy.CopySite`. +2. Build the plan from the readers of each destination: + * merge what no device work touches before the graph replay; + * anchor a `flush_after` barrier on each copy that an eager reader consumes; + * set the plan as the model's `copy_plan`. +3. Let each buffer owner skip its unchanged uploads on its own, with a `DeviceCopyBuffer` that has the copy's site. +4. Turn off the model's `RuntimeHooks` step-input pack switches (`pack_step_inputs`, `skip_unchanged_h2d` and the + `pack_*` options). The engine refuses a model that declares a plan and also turns these on (`check_one_copy_path`). +5. Run the model's tests with `TLLM_DEVICE_COPY_CHECK=1`, and compare its step time with the step-input pack it + replaces. + +## Coexistence with the step-input pack + +The copy plan replaces an older mechanism: +* the step-input pack, `pyexecutor/step_inputs.py`; +* the skip cache, `pyexecutor/h2d_cache.py`; +* `tensorrt_llm/_device_copy.DeviceCopyBuffer`; +* the `RuntimeHooks` switches that turn them on. + +Both mechanisms are supported while models move over: + +* A model that declares no copy plan has no copier, and its copies run exactly as before. A shared call site that has + a `CopySite` takes its original path whenever no copier is active (`device_copy.current()` is None). +* A model that declares a plan has a copier. The shared sites route through it, and the step-input pack stays off. +* A model uses one mechanism for all of its step's copies, never both. + +The rules while both exist: + +* New per-step copies in runtime code go through `device_copy` with a named site. +* Do not add new users of `step_inputs`, `h2d_cache`, `tensorrt_llm/_device_copy` or the `RuntimeHooks` step-input + pack switches. Existing users keep working until their model moves to a copy plan. +* When a shared call site gains a `CopySite`, keep its original path for engines without a copier, so that models + without a plan see no behavior or performance change. +* Once every model has moved, a follow-up change removes the step-input pack, the skip cache and their `RuntimeHooks` + switches. + +The same approach applies to any shared runtime mechanism that a new one replaces: +1. Add the new mechanism beside the old one, opt in and off by default. +2. Move the models in follow-up changes. +3. Remove the old mechanism last. diff --git a/docs/source/index.rst b/docs/source/index.rst index 8a801c819526..3cfa3484dce9 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -117,6 +117,7 @@ Welcome to TensorRT LLM's Documentation! developer-guide/kv-cache-cold-page-codec.md developer-guide/kv-cache-compression-development.md developer-guide/sparse-attention-development-guide.md + developer-guide/device-copy-plan.md developer-guide/telemetry.md diff --git a/tensorrt_llm/_torch/attention/backends/trtllm.py b/tensorrt_llm/_torch/attention/backends/trtllm.py index a0f5656380d1..63c8eb92d6a3 100644 --- a/tensorrt_llm/_torch/attention/backends/trtllm.py +++ b/tensorrt_llm/_torch/attention/backends/trtllm.py @@ -52,8 +52,9 @@ from ...distributed import cp_allgather from ...kimi_k3_cache_policy import get_kimi_k3_fp8_kv_layer_ids, is_kimi_k3 from ...locality_domain.runtime import LocalityDomainRuntime -from ...pyexecutor import h2d_cache, step_inputs +from ...pyexecutor import device_copy, h2d_cache, step_inputs from ...pyexecutor.config_utils import is_mla +from ...pyexecutor.device_copy import CopySite from ...pyexecutor.h2d_cache import copy_h2d_pool_shared from ...utils import (compute_swizzled_sf_shape, get_global_attrs, get_model_extra_attrs, helix_local_len_tensor) @@ -1366,7 +1367,14 @@ def prepare(self) -> None: device='cpu', ) self.prompt_lens_cpu[:self.num_seqs].copy_(prompt_lens) - if not self._upload_to_device( + if device_copy.current() is not None: + # The model declares a copy plan, which owns the engine's copies. Never + # skipped: CUDA graph metadata of every graph key takes this buffer from + # one pool, so no host record proves its device contents. + device_copy.copy(CopySite.ATTN_PROMPT_LENS, + self.prompt_lens_cuda[:self.num_seqs], + self.prompt_lens_cpu[:self.num_seqs]) + elif not self._upload_to_device( "prompt_lens", self.prompt_lens_cuda[:self.num_seqs], self.prompt_lens_cpu[:self.num_seqs], h2d_cache.enabled()): copy_h2d_pool_shared(self.prompt_lens_cuda[:self.num_seqs], @@ -1426,7 +1434,11 @@ def prepare(self) -> None: # the sequence length including the cached tokens and the input tokens. self.kv_lens[:self.num_seqs].copy_( kv_lens + self.kv_cache_params.num_extra_kv_tokens) - if not self._upload_to_device( + if device_copy.current() is not None: + device_copy.copy(CopySite.ATTN_KV_LENS, + self.kv_lens_cuda[:self.num_seqs], + maybe_pin_memory(kv_lens[:self.num_seqs])) + elif not self._upload_to_device( "kv_lens", self.kv_lens_cuda[:self.num_seqs], kv_lens[:self.num_seqs], step_inputs.is_open()): step_inputs.copy_(self.kv_lens_cuda[:self.num_seqs], @@ -1777,7 +1789,11 @@ def prepare_encoder_decoder_from_precomputed_lengths( f"({self.kv_cache_manager.max_seq_len}).") num_seqs = self.num_seqs - copy_h2d_pool_shared(self.prompt_lens_cuda[:num_seqs], prompt_lens) + if device_copy.current() is not None: + device_copy.copy(CopySite.ATTN_PROMPT_LENS, + self.prompt_lens_cuda[:num_seqs], prompt_lens) + else: + copy_h2d_pool_shared(self.prompt_lens_cuda[:num_seqs], prompt_lens) self.kv_lens_cuda[:num_seqs].copy_(kv_lens, non_blocking=True) self.host_total_kv_lens[0] = context_kv_tokens self.host_total_kv_lens[1] = generation_kv_tokens diff --git a/tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py b/tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py index aa70e7de9265..24f1743a3bdd 100644 --- a/tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py +++ b/tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py @@ -31,7 +31,8 @@ from ..speculative.spec_sampler_base import SampleStateTensorsSpec from ..speculative.utils import get_draft_kv_cache_manager from ..utils import decode_stream_priority, make_weak_ref, piecewise_cuda_graph -from . import step_inputs +from . import device_copy, step_inputs +from .device_copy import CopySite from .llm_request import LlmRequest, get_draft_token_length from .replay_inputs import stage_replay_inputs from .resource_manager import (BaseResourceManager, NoFreeSlotsError, @@ -363,6 +364,9 @@ def __init__(self, config: CUDAGraphRunnerConfig): self.graphs: Dict[KeyType, torch.cuda.CUDAGraph] = {} # Graphs captured with a step-input pack's prologue, and that pack (step_inputs.capture_prologue). self._step_input_prologue_keys: Dict[KeyType, Any] = {} + # Graphs captured with a copier's in-graph final flush prologue, and that copier + # (device_copy.capture_prologue). + self._prologue_copiers: Dict[KeyType, Any] = {} self.graph_outputs: Dict[KeyType, Callable[[], Optional[torch.Tensor]]] = {} self.graph_metadata: Dict[KeyType, Dict[str, Any]] = {} @@ -1429,6 +1433,10 @@ def run_warmup(): step_inputs.capture_prologue()) if prologue_pack is not None: self._step_input_prologue_keys[key] = prologue_pack + prologue_copier = (None if self._pre_replay_hooks else + device_copy.capture_prologue()) + if prologue_copier is not None: + self._prologue_copiers[key] = prologue_copier output = forward_fn(capture_inputs) if len(HOSTFUNC_USER_DATA_HANDLES) > num_hostfuncs: self._hostfunc_graph_keys.add(key) @@ -1443,6 +1451,16 @@ def run_warmup(): self.memory_pool = graph.pool() return graph_output + @staticmethod + def _copy_static_input(site: CopySite, dst: torch.Tensor, + src: torch.Tensor) -> None: + """Copy a step input into a graph's static input: through ``site`` when the model + declares a copy plan (an active copier), else through the step-input pack.""" + if device_copy.current() is not None: + device_copy.copy(site, dst, src) + else: + step_inputs.copy_(dst, src) + def replay( self, key: KeyType, @@ -1498,9 +1516,11 @@ def replay( static_tensors["position_ids"]) hook_positions = self._pre_replay_position_ids[..., :seqlen] # One launch copies the ids and positions and writes the hooks' positions, - # unless an open step-input pack still defers copies into its sources. + # unless an open step-input pack or copy-plan pack still defers copies into + # their sources. staged = (self.config.stage_decode_inputs and copy_input_ids and not self.config.use_mrope and not step_inputs.is_open() + and not device_copy.is_open() and tuple(position_ids.shape) == (1, seqlen) and stage_replay_inputs( static_input_ids, input_ids, @@ -1508,7 +1528,8 @@ def replay( hook_positions, None if hook_positions is None else position_id_offsets[:seqlen])) if copy_input_ids and not staged: - step_inputs.copy_(static_input_ids, input_ids) + self._copy_static_input(CopySite.GRAPH_INPUT_IDS, static_input_ids, + input_ids) if self.config.use_mrope: expected_position_ids_shape = (3, 1, seqlen) @@ -1559,7 +1580,8 @@ def replay( static_position_ids = static_tensors["position_ids"][:, :seqlen] if (not prestaged and not staged and position_ids.data_ptr() != static_position_ids.data_ptr()): - step_inputs.copy_(static_position_ids, position_ids) + self._copy_static_input(CopySite.GRAPH_POSITION_IDS, + static_position_ids, position_ids) num_encoder_tokens = key.num_encoder_tokens if num_encoder_tokens: @@ -1593,6 +1615,13 @@ def replay( if prologue_pack is not None and not captured_pack_active: # Replayed outside the pack's context: nothing filled its window. prologue_pack.quiesce_graph_window() + # The same for the copy plan's merged copies (a model that declares a plan). + prologue_copier = self._prologue_copiers.get(key) + captured_copier_active = (prologue_copier is not None + and device_copy.is_active(prologue_copier)) + device_copy.flush(in_graph=captured_copier_active) + if prologue_copier is not None and not captured_copier_active: + prologue_copier.quiesce_graph_window() if self._pre_replay_hooks: hook_tensors = static_tensors if hook_positions is not None: @@ -1609,6 +1638,8 @@ def replay( finally: if prologue_pack is not None: prologue_pack.on_graph_replay() + if prologue_copier is not None: + prologue_copier.on_graph_replay() if key in self._hostfunc_graph_keys: record_hostfunc_graph_replay() output_ref = self.graph_outputs[key] @@ -2219,6 +2250,7 @@ def clear(self, *, release_nccl_window_owners: bool = True): self._hostfunc_graph_keys.clear() self._pre_replay_position_ids = None self._step_input_prologue_keys.clear() + self._prologue_copiers.clear() self.reset_padding_dummies() del self.memory_pool self.memory_pool = None diff --git a/tensorrt_llm/_torch/pyexecutor/device_copy.py b/tensorrt_llm/_torch/pyexecutor/device_copy.py new file mode 100644 index 000000000000..b2c69f33fe78 --- /dev/null +++ b/tensorrt_llm/_torch/pyexecutor/device_copy.py @@ -0,0 +1,1284 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""A step's small copies into graph-stable device buffers, merged as the model's copy plan says. + +Between two decode CUDA graph replays, the next step's inputs and metadata reach their device buffers as a chain of tiny +copies (positions, KV lengths, page tables, the graph runner's static inputs). Each one is a separate operation on the +stream the next graph waits for, so the GPU idles for the whole chain. + +Every such copy in the runtime names its call site with a ``CopySite`` and goes through ``copy`` (or ``fill_tail``). +Two questions decide what happens to it, and two different owners answer them: + +* **When** a copy must land is the copy plan's business. It depends on the destination's readers, which the model + chose, so the model declares a ``CopyPlan`` (``copy_plan`` on the model): + + - ``merge``: the sites whose copies are staged into the step's pack. A host-to-device copy snapshots its host bytes + into pinned staging and a device-to-device copy records its (dst, src); a flush issues one upload of + [descriptors | data] and one ``trtllm::step_inputs_scatter`` launch, which writes every staged host copy and then + every device copy. A merged copy lands before the next barrier, and at the latest at the graph replay, the step's + final barrier. A pack is opened only on a step that replays an already captured graph, so it always reaches it. + - ``flush_after`` (a subset of ``merge``): barrier sites. When a call through one of them stages its copy, the pack + is flushed right after it, so that copy has landed before anything that runs next. Only the anchor's own write + is guaranteed: a reader that needs other merged data anchors on those sites too. + + - ``parallel_scatter`` and ``final_flush=InGraph(...)``: how a segment is issued. With ``parallel_scatter`` an + order-free segment (see ``_parallel_ok``) is scattered with one program per copy. With an ``InGraph`` final flush + every decode graph is captured with a prologue that issues a pinned window, and the final flush before a replay + only fills that window, so no operation is left between the decode graphs. Barrier segments stay eager: only the + last segment rides the window, and only when it fits and is order-free. + + Listing a site in ``merge`` asserts that between the call and its barrier no device work reads or writes the + destination, nor, for a device-to-device copy, writes the source (which is read at the flush). A site in neither + set, and every site of an engine whose model declares no plan, is ``dst.copy_(src, non_blocking=True)``. An eager + copy or fill issued through this module while copies are staged issues them first, so the copies land in call order + whichever sites the plan merges; only device work outside this module needs a ``flush_after`` anchor. +* **Whether** a copy is needed at all is the buffer owner's business. It depends on the values and on who else writes + the buffer, which only the owner knows: the owner passes ``unchanged=True`` when the destination already holds what + the call would write, and the call is not issued. ``DeviceCopyBuffer`` is that owner-side record for one buffer and + one site: ``update`` compares the host values with what its last issued copy wrote, issues the copy through its site + otherwise, records only a copy that was issued, and passes ``on_drop`` so a merged copy that never lands (its step + failed and the pack was discarded) invalidates the record. ``unchanged=True`` is refused during CUDA graph capture: + a captured copy reruns on every replay. + +Model-specific backends name their own sites with their own ``CopySiteEnum``. Adding a model: declare the plan from +the readers of each destination (merge what no device work touches before the replay, and anchor a barrier on each +copy that has an eager reader), set it as the model's ``copy_plan``, and let each buffer owner decide on its own whether +its uploads may be skipped (a ``DeviceCopyBuffer`` with the copy's site). + +This is the runtime's mechanism for per-step device copies going forward; it is opt-in and coexists with the older one. +An engine whose model declares no plan has no copier, and its copies run exactly as before: the shared call sites that +route through a ``CopySite`` take their original path whenever no copier is active (``current()`` is None), including +the step-input pack and skip cache that ``RuntimeHooks`` turns on (``step_inputs`` / ``h2d_cache``). A model moves to +the copy plan as a whole: an engine refuses a model that declares a plan and also turns on those ``RuntimeHooks`` +switches (``pack_step_inputs``, ``skip_unchanged_h2d``). New copy paths go through this module; the older modules stay +until every model has moved. + +``CHECK_MODE`` verifies the assertions at runtime, at the cost of a device synchronization per check: a write of a +merged destination before its flush, and an ``unchanged`` copy or fill (including every copy a ``DeviceCopyBuffer`` +skips) whose destination does not hold the values. A read of a merged destination before its barrier is caught only +where the reader asserts it with ``expect_landed`` under ``if device_copy.CHECK_MODE:``; elsewhere an early reader sees +the poison bytes instead and fails downstream. It is read once at import from ``TLLM_DEVICE_COPY_CHECK=1``, or from +``TLLM_DEBUG_MODE=1``, which turned on the verification of ``DeviceCopyBuffer``'s skipped copies before the two were +one switch. + +Each engine with a plan owns one ``DeviceCopier`` and makes it the active one for its forward (``active``); the module +functions act on the calling context's copier, so engines on other threads or devices never share one. The attention +and speculative metadata that copy have no engine to ask, which is why the copier travels in the context. +""" + +from __future__ import annotations + +import contextlib +import os +from contextvars import ContextVar +from dataclasses import dataclass +from enum import Enum +from typing import Callable, Iterator, Optional, Protocol + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._utils import prefer_pinned + +CHECK_ENV = "TLLM_DEVICE_COPY_CHECK" +#: Whether the copies' assertions are verified at runtime (``TLLM_DEVICE_COPY_CHECK=1``, or ``TLLM_DEBUG_MODE=1``), read +#: once at import. +CHECK_MODE: bool = ( + os.environ.get(CHECK_ENV, "0") == "1" or os.environ.get("TLLM_DEBUG_MODE", "")[0:1] == "1" +) +# Byte written into a merged copy's destination in check mode, so an early reader sees garbage and an early writer +# is caught at the flush. +_POISON = 0xA5 + + +class Direction(Enum): + H2D = "h2d" + D2D = "d2d" + # A device-side fill of a buffer's unused tail. + FILL = "fill" + + +class CopySiteEnum(Enum): + """Base of the copy-site enums: each member is ``(key, direction)``, the key unique across all enums.""" + + def __init__(self, key: str, direction: Direction) -> None: + self.key = key + self.direction = direction + + +class CopySite(CopySiteEnum): + """Copy sites in shared runtime code.""" + + # CUDAGraphRunner.replay: the step's input ids / position ids into the graph's static inputs. + GRAPH_INPUT_IDS = ("graph.input_ids", Direction.D2D) + GRAPH_POSITION_IDS = ("graph.position_ids", Direction.D2D) + # Decoder inputs: host position ids. + INPUT_POSITION_IDS = ("inputs.position_ids", Direction.H2D) + # TrtllmAttentionMetadata. + ATTN_KV_LENS = ("attn.kv_lens", Direction.H2D) + ATTN_PROMPT_LENS = ("attn.prompt_lens", Direction.H2D) + # SpecMetadata: the one-model speculative sampler's per-step Philox seeds and offsets (first read by the forward). + SPEC_RNG_REQUEST_SEEDS = ("spec.rng_request_seeds", Direction.H2D) + SPEC_RNG_REQUEST_OFFSETS = ("spec.rng_request_offsets", Direction.H2D) + SPEC_RNG_SEEDS = ("spec.rng_seeds", Direction.H2D) + SPEC_RNG_OFFSETS = ("spec.rng_offsets", Direction.H2D) + + +class PrologueOp(Protocol): + """A model's one-launch in-graph prologue (``InGraph.prologue_op``). + + Called inside the capture as ``op(window, epoch_or_None, max_desc)``: it writes the window's host copies, then its + device copies (at most ``max_desc`` of each), zeroes the window's header and, with an epoch, increments + ``epoch[0]`` once. It writes exactly what the default prologue (two scatter launches and a header fill) writes. + """ + + def __call__( + self, window: torch.Tensor, epoch: Optional[torch.Tensor], max_desc: int + ) -> None: ... + + +@dataclass(frozen=True) +class InGraph: + """The step's final barrier issued by the replayed decode graph itself (``CopyPlan.final_flush``). + + Every decode graph is captured with a prologue that scatters a fixed window, and the final flush before a replay + only fills that window on the host: no operation is left between the decode graphs. Only the step's last segment + rides the window, and only when it fits and is order-free (see ``_parallel_ok``); otherwise it is issued eagerly + and the window is left empty. A graph whose runner has pre-replay hooks is captured without a prologue, since the + hooks read the static inputs before the replay. + + ``early_copy``: the window is uploaded on a side copy stream into a device buffer of the prologue's own while the + previous graph may still run; the prologue then waits for that upload and scatters device to device. + ``prologue_op``: the model's one-launch prologue (needs ``early_copy``). ``step_epoch``: the prologue increments + one int64 device counter once per replay (``step_epoch()``), never reset, so kernels later in the graph can tag + per-step device flags with it instead of clearing them between steps. + """ + + early_copy: bool = False + prologue_op: Optional[PrologueOp] = None + step_epoch: bool = False + + def __post_init__(self) -> None: + if self.prologue_op is not None and not self.early_copy: + raise ValueError( + "InGraph.prologue_op scatters the early-copied window, so it needs early_copy=True" + ) + + +@dataclass(frozen=True) +class CopyPlan: + """When a model's per-step copies land. Empty by default: every copy is plain. + + ``merge``: sites whose copies are staged into the step's pack and land before the next barrier (see the module + docstring). ``flush_after``: barrier sites, a subset of ``merge``. A call through one that stages its copy flushes + the pack right after it; a call whose copy is unchanged, or runs eagerly, has nothing of its own to land and does + not flush. ``changed_span`` (a subset of ``merge``): sites whose ``DeviceCopyBuffer.update(..., changed_span=True)`` + uploads may stage only each row's changed span (see there). ``parallel_scatter``: a segment whose copies are + order-free is scattered with one program per copy instead of the serial one-CTA scatter. ``final_flush``: ``None`` + issues the step's last segment eagerly before the replay; ``InGraph(...)`` lets the replayed graph issue it (needs + ``parallel_scatter``). + """ + + merge: frozenset[CopySiteEnum] = frozenset() + flush_after: frozenset[CopySiteEnum] = frozenset() + changed_span: frozenset[CopySiteEnum] = frozenset() + parallel_scatter: bool = False + final_flush: Optional[InGraph] = None + + def __post_init__(self) -> None: + for name in ("merge", "flush_after", "changed_span"): + object.__setattr__(self, name, frozenset(getattr(self, name))) + for site in self.merge | self.flush_after | self.changed_span: + if not isinstance(site, CopySiteEnum): + raise TypeError(f"CopyPlan sites must be CopySiteEnum members, got {site!r}") + fills = [site for site in self.merge if site.direction is Direction.FILL] + if fills: + raise ValueError(f"fill sites run on the device and cannot be merged: {fills}") + unmerged = self.flush_after - self.merge + if unmerged: + raise ValueError( + "a flush_after barrier fires only when its own copy was merged, so it must be in merge: " + f"{sorted(unmerged, key=str)}" + ) + unmerged_spans = self.changed_span - self.merge + if unmerged_spans: + raise ValueError( + "a changed-span upload stages several copies into the pack, so its site must be in merge: " + f"{sorted(unmerged_spans, key=str)}" + ) + if self.final_flush is not None: + if not isinstance(self.final_flush, InGraph): + raise TypeError( + f"CopyPlan.final_flush must be None or InGraph, got {self.final_flush!r}" + ) + if not self.parallel_scatter: + raise ValueError( + "an InGraph final flush scatters in parallel, so it needs parallel_scatter=True" + ) + if not self.merge: + raise ValueError( + "an InGraph final flush issues merged copies, so the plan must merge some" + ) + + def __bool__(self) -> bool: + return bool(self.merge) + + +def copy_plan_of(model: object) -> CopyPlan: + """The plan ``model`` declares as ``copy_plan``, or the empty plan.""" + plan = getattr(model, "copy_plan", None) + if plan is None: + return CopyPlan() + if not isinstance(plan, CopyPlan): + raise TypeError( + f"{type(model).__name__}.copy_plan must be a CopyPlan, got {type(plan).__name__}" + ) + return plan + + +def _byte_range(t: torch.Tensor) -> tuple[int, int]: + """[first, last + 1) device bytes ``t`` covers.""" + if t.numel() == 0: + return (t.data_ptr(), t.data_ptr()) + if t.is_contiguous(): + return (t.data_ptr(), t.data_ptr() + t.numel() * t.element_size()) + span = sum((size - 1) * stride for size, stride in zip(t.shape, t.stride())) + 1 + return (t.data_ptr(), t.data_ptr() + span * t.element_size()) + + +def _overlaps(r: tuple[int, int], ranges: list[tuple[int, int]]) -> bool: + return any(r[0] < hi and lo < r[1] for lo, hi in ranges) + + +def _capturing(t: torch.Tensor) -> bool: + """Whether a copy into ``t`` would be captured; a host tensor never is, and never initializes CUDA.""" + return t.is_cuda and torch.cuda.is_current_stream_capturing() + + +def _site_name(site: CopySiteEnum) -> str: + return f"{type(site).__name__}.{site.name}" + + +# One compiled variant for every descriptor offset: specializing ``first`` (== 1, divisible by 16) would compile a +# new kernel when a step's segment first holds a new number of copies, after the warm-up. +@triton.jit(do_not_specialize=["first"]) +def _parallel_scatter_kernel( + stage_u8, stage_i64, first, DEVICE_SRC: tl.constexpr, BLOCK: tl.constexpr +): + """Program i writes copy ``first + i`` ([dst, src, bytes] descriptors after the 2-word header). ``src`` is an offset + into the staged segment for a host copy, a device address for a device copy (``DEVICE_SRC``).""" + d = 2 + 3 * (first + tl.program_id(0)) + dst = tl.load(stage_i64 + d).to(tl.pointer_type(tl.uint8)) + src_word = tl.load(stage_i64 + d + 1) + if DEVICE_SRC: + src = src_word.to(tl.pointer_type(tl.uint8)) + else: + src = stage_u8 + src_word + nbytes = tl.load(stage_i64 + d + 2) + for start in range(0, nbytes, BLOCK): + o = start + tl.arange(0, BLOCK) + tl.store(dst + o, tl.load(src + o, mask=o < nbytes), mask=o < nbytes) + + +@triton.jit +def _graph_scatter_kernel( + stage_u8, + stage_i64, + epoch, + DEVICE_SRC: tl.constexpr, + BLOCK: tl.constexpr, + BUMP_EPOCH: tl.constexpr, +): + """Descriptor slot i of the host copies (``DEVICE_SRC`` False) or of the device copies, with the counts read from + the staged header; a slot past its count writes nothing. ``BUMP_EPOCH``: program 0 also increments ``epoch[0]``.""" + i = tl.program_id(0) + if BUMP_EPOCH: + if i == 0: + tl.store(epoch, tl.load(epoch) + 1) + num_h2d = tl.load(stage_i64) + if DEVICE_SRC: + count = tl.load(stage_i64 + 1) + d = 2 + 3 * (num_h2d + i) + else: + count = num_h2d + d = 2 + 3 * i + if i < count: + dst = tl.load(stage_i64 + d).to(tl.pointer_type(tl.uint8)) + src_word = tl.load(stage_i64 + d + 1) + if DEVICE_SRC: + src = src_word.to(tl.pointer_type(tl.uint8)) + else: + src = stage_u8 + src_word + nbytes = tl.load(stage_i64 + d + 2) + for start in range(0, nbytes, BLOCK): + o = start + tl.arange(0, BLOCK) + tl.store(dst + o, tl.load(src + o, mask=o < nbytes), mask=o < nbytes) + + +def _disjoint(ranges: list[tuple[int, int]]) -> bool: + """Whether the [start, start + bytes) ranges are pairwise disjoint.""" + end = -1 + for start, nbytes in sorted(ranges): + if start < end: + return False + end = start + nbytes + return True + + +def _parallel_ok(h2d: list[tuple[int, int, int]], d2d: list[tuple[int, int, int]]) -> bool: + """Whether issuing the host copies in parallel and then the device copies in parallel writes what the serial + scatter (every host copy in order, then every device copy in order) writes: the host copies' destinations are + disjoint, and the device copies' destinations are disjoint and none of them is a device copy's source (a host copy + may still feed a device copy, as the launches run in order).""" + if not _disjoint([(dst, n) for dst, _, n in h2d]): + return False + if not d2d: + return True + dsts = [(dst, n) for dst, _, n in d2d] + if not _disjoint(dsts): + return False + return all(_disjoint(dsts + [(src, n)]) for _, src, n in d2d) + + +class _Pack: + """One engine's pack of merged copies (see the module docstring). + + A step's pack lives in one slot of a ring of pinned staging buffers and in one device staging buffer. Each barrier + issues the copies staged since the previous one as a segment ([header | descriptors | data], aligned) appended + after the previous segment in the same slot, with its own upload and scatter launch. The ring advances once per + step, at ``begin``, which waits only for the upload that last used the slot RING steps ago. + + With an ``InGraph`` final flush, the step's last segment is written instead into a pinned window (``ghost``) that + the prologue of every captured decode graph issues (``capture_prologue``): a host copy of the window into the + device staging buffer and the graph scatter, or, with ``early_copy``, a wait on the window's side-stream upload + into the prologue's own device buffer (``gdev``) and the scatter from there. A segment's descriptors address its + data relative to its own start, so the window is the segment's bytes as they are. + """ + + CAPACITY = 1 << 16 + MAX_DESC = 256 + # Pinned staging buffers in rotation, one per step: a step stages without waiting on the previous steps' uploads. + RING = 4 + HEADER_BYTES = 16 + DESC_BYTES = 24 + ALIGN = 16 + + # The pinned window a graph's prologue issues (InGraph): the header, the descriptors and up to ~10 KB of data. + GRAPH_WINDOW = 1 << 14 + + def __init__( + self, + device: torch.device, + check: bool, + parallel_scatter: bool = False, + in_graph: Optional[InGraph] = None, + ) -> None: + self.host = [ + torch.empty(self.CAPACITY, dtype=torch.uint8, pin_memory=prefer_pinned()) + for _ in range(self.RING) + ] + self.events: list[Optional[torch.cuda.Event]] = [None] * self.RING + self.dev = torch.empty(self.CAPACITY, dtype=torch.uint8, device=device) + # A segment's data starts after its header and descriptor area. + self.data_start = self.HEADER_BYTES + self.DESC_BYTES * self.MAX_DESC + self.slot = self.RING - 1 + self.open = False + self.check = check + # First byte of the current segment in the slot. + self.seg = 0 + self.h2d: list[tuple[int, int, int]] = [] + self.d2d: list[tuple[int, int, int]] = [] + # Bytes every staged device copy of the segment reads or writes: a later host copy into them would land out + # of order (the scatter writes the host copies first). + self.d2d_ranges: list[tuple[int, int]] = [] + # Check mode: bytes every staged copy of the segment writes. + self.dst_ranges: list[tuple[int, int]] = [] + # Every tensor a staged copy reads or writes, kept alive until its segment is issued. + self.staged: list[torch.Tensor] = [] + # The tensors of the segment in the graph window, kept alive until the replay that issues them is queued. + self.graph_staged: list[torch.Tensor] = [] + # The callers' on_drop callbacks of the segment's copies, called if the pack is discarded. + self.drops: list[Callable[[], None]] = [] + # Check mode: (site, destination) of every staged copy, and (site, source, snapshot) of every device copy. + self.poisoned: list[tuple[CopySiteEnum, torch.Tensor]] = [] + self.sources: list[tuple[CopySiteEnum, torch.Tensor, torch.Tensor]] = [] + self.used = self.seg + self.data_start + self.parallel_scatter = parallel_scatter + self.in_graph = in_graph + self.early_copy = in_graph is not None and in_graph.early_copy + self.prologue_op = in_graph.prologue_op if in_graph is not None else None + # The step epoch (InGraph.step_epoch); None without it. + self.epoch: Optional[torch.Tensor] = ( + torch.zeros(1, dtype=torch.int64, device=device) + if in_graph is not None and in_graph.step_epoch + else None + ) + if self.early_copy: + # The prologue's own device window: an eager segment (self.dev) can never leave a header in it. + self.gdev = torch.zeros(self.GRAPH_WINDOW, dtype=torch.uint8, device=device) + self.copy_stream = torch.cuda.Stream(device=device) + # Recorded by the side upload once it landed (the prologue waits on it) ... + self.staged_ready = torch.cuda.Event(external=True) + # ... and by every replayed prologue after its scatter (the next upload waits on it). + self.prologue_done = torch.cuda.Event(external=True) + # A CUDA event is created on its first record, and a wait on one never created is skipped: a capture + # would then leave the prologue's wait out of the graph. Create both now. + self.staged_ready.record(self.copy_stream) + self.prologue_done.record(self.copy_stream) + if in_graph is not None: + self.ghost = torch.zeros( + self.GRAPH_WINDOW, dtype=torch.uint8, pin_memory=prefer_pinned() + ) + # Recorded by every replayed prologue right after it copied the window (an event node). + self.ghost_free = torch.cuda.Event(external=True) + if parallel_scatter: + self._warm_up(device) + if in_graph is not None: + self._warm_up_graph_kernel() + + def _warm_up(self, device: torch.device) -> None: + """Compile the parallel scatter now, not on a serving step's flush.""" + dst = torch.zeros(8, dtype=torch.int32, device=device) + check, self.check = self.check, False + try: + self.begin() + self.stage(None, dst[:4], torch.arange(4, dtype=torch.int32)) + self.stage(None, dst[4:], dst[:4]) + self.flush() + finally: + self.check = check + torch.cuda.synchronize(device) + # The next step starts in a fresh slot and segment. + self.seg = 0 + + def _warm_up_graph_kernel(self) -> None: + """Compile the prologue's scatter now (a zero-count header: it writes nothing).""" + self.dev[: self.HEADER_BYTES].zero_() + if self.prologue_op is not None: + # The window it is captured with; its header is zero until the first upload. + self.prologue_op(self.gdev, self.epoch, self.MAX_DESC) + else: + self._launch_graph_scatter(self.dev) + torch.cuda.synchronize(self.dev.device) + + def _launch_graph_scatter(self, window: torch.Tensor) -> None: + stage_i64 = window.view(torch.int64) + for device_src in (False, True): + bump = self.epoch is not None and not device_src + _graph_scatter_kernel[(self.MAX_DESC,)]( + window, + stage_i64, + self.epoch if bump else stage_i64, + DEVICE_SRC=device_src, + BLOCK=256, + BUMP_EPOCH=bump, + num_warps=1, + ) + + def capture_prologue(self) -> bool: + """Inside a graph capture: the prologue that issues the window ``flush(in_graph=True)`` fills. Returns whether + the graph has it.""" + if self.in_graph is None: + return False + if self.early_copy: + torch.cuda.current_stream().wait_event(self.staged_ready) + if self.prologue_op is not None: + self.prologue_op(self.gdev, self.epoch, self.MAX_DESC) + else: + self._launch_graph_scatter(self.gdev) + # Consumed: a replay with no new upload scatters nothing. + self.gdev[: self.HEADER_BYTES].zero_() + self.prologue_done.record() + return True + self.dev[: self.GRAPH_WINDOW].copy_(self.ghost, non_blocking=True) + self.ghost_free.record() + self._launch_graph_scatter(self.dev) + return True + + def _upload_window(self, nbytes: int) -> None: + """early_copy: upload the filled pinned window into gdev on the side stream, once the last replayed prologue + has consumed gdev; the window is fully written on the host before this is enqueued.""" + stream = self.copy_stream + stream.wait_event(self.prologue_done) + with torch.cuda.stream(stream): + self.gdev[:nbytes].copy_(self.ghost[:nbytes], non_blocking=True) + self.ghost_free.record(stream) + self.staged_ready.record(stream) + + def _ghost_writable(self) -> None: + # The last replayed prologue (or side upload) has copied the window (a no-op before the first replay). + self.ghost_free.synchronize() + + def quiesce_graph_window(self) -> None: + """Before replaying a graph with the prologue without filling the window: a zero-count window, so it writes + nothing.""" + if self.in_graph is not None and self.ghost[: self.HEADER_BYTES].any(): + self._ghost_writable() + self.ghost[: self.HEADER_BYTES].zero_() + if self.early_copy: + # A window uploaded but never replayed would still sit in gdev: overwrite its header. + self._upload_window(self.HEADER_BYTES) + + def _fill_graph_window(self) -> bool: + """Write the open segment into the pinned window the replay's prologue issues; False when it does not fit or + is not order-free (the caller issues it eagerly).""" + nbytes = self.used - self.seg + if nbytes > self.GRAPH_WINDOW or not _parallel_ok(self.h2d, self.d2d): + return False + desc = [len(self.h2d), len(self.d2d)] + [ + value for copy in self.h2d + self.d2d for value in copy + ] + self._ghost_writable() + self.ghost[:nbytes].copy_(self.host[self.slot][self.seg : self.used]) + self.ghost[: 8 * len(desc)].view(torch.int64).copy_(torch.tensor(desc, dtype=torch.int64)) + if self.early_copy: + self._upload_window(nbytes) + return True + + def on_graph_replay(self) -> None: + """Release the window's operands once the graph reading their addresses has been queued.""" + if self.graph_staged: + stream = torch.cuda.current_stream(self.dev.device) + for tensor in self.graph_staged: + tensor.record_stream(stream) + self.graph_staged.clear() + + def begin(self) -> None: + if self.open: + self.flush() + self.slot = (self.slot + 1) % self.RING + event = self.events[self.slot] + if event is not None: + # The uploads that last read this pinned buffer have run. + event.synchronize() + self.seg = 0 + self._clear() + self.open = True + + def _clear(self) -> None: + self.h2d.clear() + self.d2d.clear() + self.d2d_ranges.clear() + self.dst_ranges.clear() + self.staged.clear() + self.drops.clear() + self.poisoned.clear() + self.sources.clear() + self.used = self.seg + self.data_start + + def writes(self, t: torch.Tensor) -> bool: + """Whether a staged copy writes any byte of ``t`` (check mode only).""" + return _overlaps(_byte_range(t), self.dst_ranges) + + def pending_sites(self, t: torch.Tensor) -> list[CopySiteEnum]: + """The sites of staged, not yet issued copies that write any byte of ``t`` (check mode only).""" + r = _byte_range(t) + return [site for site, dst in self.poisoned if _overlaps(r, [_byte_range(dst)])] + + def stage( + self, + site: CopySiteEnum, + dst: torch.Tensor, + src: torch.Tensor, + on_drop: Optional[Callable[[], None]] = None, + ) -> bool: + if not dst.is_cuda or dst.numel() != src.numel(): + return False + nbytes = dst.numel() * dst.element_size() + # Byte ranges are needed only against staged device copies, and in check mode. + dst_range = _byte_range(dst) if self.d2d_ranges or self.check or src.is_cuda else None + if not src.is_cuda: + if self.d2d_ranges and _overlaps(dst_range, self.d2d_ranges): + return False + # A contiguous destination is one descriptor; a 2-D slice with contiguous rows (a page table's leading + # rows and columns) is one per row. + if dst.is_contiguous(): + rows, row_bytes, row_stride = 1, nbytes, 0 + elif dst.dim() == 2 and dst.stride(1) == 1: + rows = dst.shape[0] + row_bytes = dst.shape[1] * dst.element_size() + row_stride = dst.stride(0) * dst.element_size() + else: + return False + offset = (self.used + self.ALIGN - 1) // self.ALIGN * self.ALIGN + if ( + offset + nbytes > self.CAPACITY + or len(self.h2d) + len(self.d2d) + rows > self.MAX_DESC + ): + return False + # A dtype change converts on the host, as copy_ would on the device. + src = src.to(dst.dtype).contiguous() + self.host[self.slot][offset : offset + nbytes].copy_(src.view(-1).view(torch.uint8)) + base = dst.data_ptr() + # The scatter reads a host copy's bytes at an offset from its segment's start. + rel = offset - self.seg + for row in range(rows): + self.h2d.append((base + row * row_stride, rel + row * row_bytes, row_bytes)) + self.staged.append(dst) + self.used = offset + nbytes + else: + if self.used > self.CAPACITY or len(self.h2d) + len(self.d2d) >= self.MAX_DESC: + return False + if not ( + src.is_cuda + and src.dtype == dst.dtype + and src.is_contiguous() + and dst.is_contiguous() + ): + return False + src_range = _byte_range(src) + if self.check and not _overlaps(src_range, self.dst_ranges): + # A source no staged copy fills must not change before the flush reads it. + self.sources.append((site, src, src.clone())) + self.d2d.append((dst.data_ptr(), src.data_ptr(), nbytes)) + self.d2d_ranges += (dst_range, src_range) + self.staged += (dst, src) + if on_drop is not None: + self.drops.append(on_drop) + if self.check: + self.dst_ranges.append(dst_range) + dst.view(torch.uint8).fill_(_POISON) + self.poisoned.append((site, dst)) + return True + + def _verify(self) -> None: + for site, dst in self.poisoned: + if not bool(dst.view(torch.uint8).eq(_POISON).all()): + raise RuntimeError( + f"{CHECK_ENV}: the destination of merged copy {_site_name(site)} was written between the " + "copy and its flush; the plan must not merge this site, or needs a barrier before that writer" + ) + for site, src, snapshot in self.sources: + if not torch.equal(src, snapshot): + raise RuntimeError( + f"{CHECK_ENV}: the source of merged device copy {_site_name(site)} changed between the copy " + "and its flush; the plan must not merge this site, or needs a barrier before that writer" + ) + + def _issue(self) -> None: + """Issue the segment staged since the last barrier and start the next one after it in the same slot.""" + if not self.h2d and not self.d2d: + return + if self.check: + self._verify() + host, seg = self.host[self.slot], self.seg + desc = [len(self.h2d), len(self.d2d)] + [ + value for copy in self.h2d + self.d2d for value in copy + ] + host[seg : seg + 8 * len(desc)].view(torch.int64).copy_( + torch.tensor(desc, dtype=torch.int64) + ) + # Descriptors and data in one upload; the gap between them is unused padding. + self.dev[seg : self.used].copy_(host[seg : self.used], non_blocking=True) + if self.parallel_scatter and _parallel_ok(self.h2d, self.d2d): + stage = self.dev[seg:] + stage_i64 = stage.view(torch.int64) + if self.h2d: + _parallel_scatter_kernel[(len(self.h2d),)]( + stage, stage_i64, 0, DEVICE_SRC=False, BLOCK=256, num_warps=2 + ) + if self.d2d: + _parallel_scatter_kernel[(len(self.d2d),)]( + stage, stage_i64, len(self.h2d), DEVICE_SRC=True, BLOCK=256, num_warps=2 + ) + else: + torch.ops.trtllm.step_inputs_scatter(self.dev[seg:]) + event = self.events[self.slot] + if event is None: + event = self.events[self.slot] = torch.cuda.Event() + event.record() + # The scatter is queued on the current stream: the caching allocator keeps the staged tensors' memory for it. + self.seg = (self.used + self.ALIGN - 1) // self.ALIGN * self.ALIGN + self._clear() + + def barrier(self) -> None: + """Issue what is staged; the pack stays open for the rest of the step.""" + if self.open: + self._issue() + + def flush(self, in_graph: bool = False) -> None: + """Issue what is staged and close the pack (the step's final barrier). + + ``in_graph``: the next replay is of a graph captured with this pack's prologue, which issues the segment if it + rides the window; otherwise the segment is issued eagerly and the window left empty. + """ + if not self.open: + if in_graph: + self.quiesce_graph_window() + return + self.open = False + if in_graph and self.in_graph is not None: + if self.check: + self._verify() + if self._fill_graph_window(): + self.graph_staged.extend(self.staged) + self._clear() + return + self.quiesce_graph_window() + self._issue() + + def discard(self) -> None: + """Close the pack without issuing it (its step failed); the unissued copies' on_drop callbacks run.""" + drops = list(self.drops) + self.open = False + self.seg = 0 + self._clear() + for on_drop in drops: + on_drop() + + +def _view_key(t: torch.Tensor) -> tuple: + return (t.storage_offset(), tuple(t.shape), tuple(t.stride()), t.dtype) + + +class DeviceCopier: + """One engine's merge pack, applied per site as its ``CopyPlan`` says. + + In check mode (``CHECK_MODE`` when the copier is built), every merged copy's destination is poisoned at the call and + verified untouched when its segment is issued, every merged device copy's source is verified unchanged then, and + every ``unchanged`` copy or fill is verified against the device contents. A failed check raises and names the site. + A device read of a merged destination before its barrier is detected only by a reader that calls + ``expect_landed``; any other early reader reads the poison. + """ + + def __init__(self, plan: CopyPlan, device: torch.device) -> None: + self.plan = plan + self.check = CHECK_MODE + self._pack = ( + _Pack(device, self.check, plan.parallel_scatter, plan.final_flush) + if plan.merge + else None + ) + # Site membership by identity: an Enum member hashes through a Python-level __hash__. + self._merge_ids = frozenset(map(id, plan.merge)) + self._barrier_ids = frozenset(map(id, plan.flush_after)) + self._changed_span_ids = frozenset(map(id, plan.changed_span)) + + @property + def merges(self) -> bool: + return self._pack is not None + + def pending_sites(self, t: torch.Tensor) -> list[CopySiteEnum]: + """Check mode: the merged sites whose staged, not yet issued copies write any byte of ``t``.""" + if self._pack is None or not self._pack.open: + return [] + return self._pack.pending_sites(t) + + def writes_pending(self, t: torch.Tensor) -> bool: + """Check mode: whether a staged, not yet issued copy writes any byte of ``t``.""" + return self._pack is not None and self._pack.open and self._pack.writes(t) + + @property + def is_open(self) -> bool: + return self._pack is not None and self._pack.open + + def merges_now(self, site: CopySiteEnum) -> bool: + """Whether a copy through ``site`` would be staged into the open pack now.""" + return ( + self._pack is not None + and self._pack.open + and id(site) in self._merge_ids + and not torch.cuda.is_current_stream_capturing() + ) + + def stages_changed_span(self, site: CopySiteEnum) -> bool: + """Whether a changed-span upload through ``site`` may stage per-row spans now (``CopyPlan.changed_span``). Not + through a barrier site: each span would flush on its own, so the whole view is staged with one barrier.""" + return ( + id(site) in self._changed_span_ids + and id(site) not in self._barrier_ids + and self.merges_now(site) + ) + + def begin(self) -> None: + if self._pack is not None: + self._pack.begin() + + def flush(self, in_graph: bool = False) -> None: + if self._pack is not None: + self._pack.flush(in_graph) + + @property + def epoch(self) -> Optional[torch.Tensor]: + """The step epoch (``InGraph.step_epoch``), or None.""" + return self._pack.epoch if self._pack is not None else None + + def capture_prologue(self) -> bool: + """Inside a graph capture: add the in-graph final flush's prologue; whether the graph has it.""" + return self._pack is not None and self._pack.capture_prologue() + + def quiesce_graph_window(self) -> None: + """Before replaying a graph with this copier's prologue outside its context: an empty window.""" + if self._pack is not None: + self._pack.quiesce_graph_window() + + def on_graph_replay(self) -> None: + """After queueing the replay of a graph with this copier's prologue: release the window's operands.""" + if self._pack is not None: + self._pack.on_graph_replay() + + def discard(self) -> None: + """Drop the open pack: the step failed, so its staged copies never land (their on_drop callbacks run).""" + if self._pack is not None: + self._pack.discard() + + def copy( + self, + site: CopySiteEnum, + dst: torch.Tensor, + src: torch.Tensor, + unchanged: bool = False, + on_drop: Optional[Callable[[], None]] = None, + snapshot: Optional[Callable[[torch.Tensor], torch.Tensor]] = None, + ) -> bool: + if site.direction is Direction.FILL: + raise ValueError(f"{_site_name(site)} is a fill site; use fill_tail") + if src.is_cuda == (site.direction is Direction.H2D): + raise ValueError( + f"{_site_name(site)} is declared {site.direction.value}, but its source is on {src.device}" + ) + if unchanged: + _check_unchanged_copy(site, dst, src, self) + return False + pack = self._pack + if ( + pack is not None + and pack.open + and id(site) in self._merge_ids + and not torch.cuda.is_current_stream_capturing() + ): + if pack.stage(site, dst, src, on_drop): + if id(site) in self._barrier_ids: + pack.barrier() + return True + # Cannot be staged: what is staged lands first, then this copy (which keeps the order). + pack.barrier() + else: + self._land_staged() + dst.copy_(src if snapshot is None else snapshot(src), non_blocking=True) + return True + + def fill_tail( + self, site: CopySiteEnum, buf: torch.Tensor, start: int, value: int, unchanged: bool = False + ) -> None: + if site.direction is not Direction.FILL: + raise ValueError(f"{_site_name(site)} is not a fill site") + if not unchanged and start < buf.shape[0]: + self._land_staged() + _fill_tail(site, buf, start, value, unchanged, self) + + def _land_staged(self) -> None: + """Before an eager copy or fill: issue what is staged, so the eager one lands after every earlier copy.""" + pack = self._pack + if ( + pack is not None + and pack.open + and (pack.h2d or pack.d2d) + and not torch.cuda.is_current_stream_capturing() + ): + pack.barrier() + + +def _check_unchanged_copy( + site: CopySiteEnum, dst: torch.Tensor, src: torch.Tensor, copier: Optional[DeviceCopier] +) -> None: + """An ``unchanged`` copy is refused during capture and, in check mode, verified against the device.""" + if _capturing(dst): + raise ValueError( + f"{_site_name(site)}: unchanged=True during CUDA graph capture; a captured copy reruns on every " + "replay, so it must be issued (and not recorded as skippable)" + ) + if CHECK_MODE and not (copier is not None and copier.writes_pending(dst)): + if not torch.equal(dst.cpu(), src.to(device="cpu", dtype=dst.dtype)): + raise RuntimeError( + f"{CHECK_ENV}: copy {_site_name(site)} was passed unchanged=True, but its destination holds other " + "values; something else writes this destination, or the owner's record is stale" + ) + + +def _fill_tail( + site: CopySiteEnum, + buf: torch.Tensor, + start: int, + value: int, + unchanged: bool, + copier: Optional[DeviceCopier], +) -> None: + if start >= buf.shape[0]: + return + if not unchanged: + buf[start:].fill_(value) + return + if _capturing(buf): + raise ValueError( + f"{_site_name(site)}: unchanged=True during CUDA graph capture; a captured fill reruns on every replay" + ) + tail = buf[start:] + if CHECK_MODE and not (copier is not None and copier.writes_pending(tail)): + if not bool(tail.eq(value).all()): + raise RuntimeError( + f"{CHECK_ENV}: tail fill {_site_name(site)} was passed unchanged=True, but the tail holds other " + "values; something else writes this tail, or the owner's record is stale" + ) + + +_active: ContextVar[Optional[DeviceCopier]] = ContextVar("trtllm_device_copier", default=None) + + +@contextlib.contextmanager +def active(copier: Optional[DeviceCopier]) -> Iterator[None]: + """Make ``copier`` the calling context's copier (None: plain copies) for the duration.""" + token = _active.set(copier) + try: + yield + finally: + _active.reset(token) + + +def current() -> Optional[DeviceCopier]: + return _active.get() + + +def copy( + site: CopySiteEnum, + dst: torch.Tensor, + src: torch.Tensor, + *, + unchanged: bool = False, + on_drop: Optional[Callable[[], None]] = None, + snapshot: Optional[Callable[[torch.Tensor], torch.Tensor]] = None, +) -> bool: + """``dst.copy_(src, non_blocking=True)``, merged as the active copier's plan says; True if issued. + + ``unchanged``: the caller's assertion that ``dst`` already holds ``src``; the copy is not issued and False is + returned. Refused during CUDA graph capture, and verified in check mode. ``on_drop``: called if the copy was merged + and its pack is then discarded without issuing it (the caller's record of the copy is then stale). ``snapshot``: + applied to ``src`` when the copy is issued eagerly, for a caller that writes ``src`` again before the copy runs (a + merged copy is snapshotted into the pack already). + """ + copier = _active.get() + if copier is not None: + return copier.copy(site, dst, src, unchanged, on_drop, snapshot) + if unchanged: + _check_unchanged_copy(site, dst, src, None) + return False + dst.copy_(src if snapshot is None else snapshot(src), non_blocking=True) + return True + + +def fill_tail( + site: CopySiteEnum, buf: torch.Tensor, start: int, value: int, *, unchanged: bool = False +) -> None: + """``buf[start:].fill_(value)``; not issued when the caller asserts ``unchanged`` (the tail already holds it). + + ``unchanged`` is refused during CUDA graph capture and verified in check mode. + """ + copier = _active.get() + if copier is not None: + copier.fill_tail(site, buf, start, value, unchanged) + return + _fill_tail(site, buf, start, value, unchanged, None) + + +def expect_landed(site: CopySiteEnum, t: torch.Tensor) -> None: + """Check mode: assert that no merged copy into ``t`` is still staged, i.e. a barrier precedes this reader. + + ``site`` names the reader's anchor (the site whose ``flush_after`` barrier should have issued the copies) in the + error. Call it only under ``if device_copy.CHECK_MODE:``; it raises when check mode is off, so an unguarded call + cannot add work to the hot path unnoticed. Without an active copier nothing is staged and it passes. + """ + if not CHECK_MODE: + raise RuntimeError( + f"expect_landed({_site_name(site)}) is a {CHECK_ENV} assertion; call it under `if device_copy.CHECK_MODE:`" + ) + copier = _active.get() + if copier is None: + return + pending = copier.pending_sites(t) + if pending: + raise RuntimeError( + f"{CHECK_ENV}: a reader anchored at {_site_name(site)} reads bytes that merged copies " + f"{sorted({_site_name(p) for p in pending})} have not written yet; the plan needs {_site_name(site)} " + "in flush_after, and its call must precede this reader" + ) + + +def begin() -> None: + """Open the active copier's pack. Call only when the step will reach ``flush`` (a captured graph's replay).""" + copier = _active.get() + if copier is not None: + copier.begin() + + +def is_open() -> bool: + copier = _active.get() + return copier is not None and copier.is_open + + +def stages_changed_span(site: CopySiteEnum) -> bool: + """Whether a changed-span upload through ``site`` may stage per-row spans now (see ``DeviceCopyBuffer.update``).""" + copier = _active.get() + return copier is not None and copier.stages_changed_span(site) + + +def flush(in_graph: bool = False) -> None: + """Issue the active open pack's last segment (one upload and one scatter launch on the current stream) and close it. + + The step's final barrier, at the graph replay. ``in_graph``: the next replay is of a graph captured with the active + copier's prologue (``capture_prologue``), which issues the segment itself when it rides the window. + """ + copier = _active.get() + if copier is not None: + copier.flush(in_graph) + + +def is_active(copier: DeviceCopier) -> bool: + """Whether ``copier`` is the calling context's copier.""" + return _active.get() is copier + + +def capture_prologue() -> Optional[DeviceCopier]: + """Inside a graph capture: add the active copier's in-graph final flush prologue (``InGraph``); the copier whose + window the graph issues, or None when the graph has no prologue.""" + copier = _active.get() + return copier if copier is not None and copier.capture_prologue() else None + + +def step_epoch() -> Optional[torch.Tensor]: + """The active copier's step epoch (an int64 [1] device tensor, ``InGraph.step_epoch``), or None.""" + copier = _active.get() + return copier.epoch if copier is not None else None + + +def discard() -> None: + """Drop the active copier's open pack (its step failed); the staged copies' on_drop callbacks run.""" + copier = _active.get() + if copier is not None: + copier.discard() + + +class DeviceCopyBuffer: + """A device buffer's record of what its last issued copy wrote, so its owner skips uploads of values it holds. + + ``site`` names the buffer's uploads (a host-to-device ``CopySiteEnum`` member): ``update`` issues them through + ``copy``, so a model's plan may merge them, and check mode verifies every skipped one. ``site=None`` makes a + record-only buffer for an owner that writes the buffer through its own path and keeps the record with ``matches`` / + ``record`` / ``invalidate``. + + The record follows the copy: it is dropped before a copy is issued (a copy that fails leaves none), written only + after the copy was issued, dropped again if a merged copy never lands (``on_drop``, the pack was discarded), and + neither consulted nor written during CUDA graph capture, where a copy is always issued: a captured copy reruns on + every replay. An owner that writes the buffer by other means calls ``invalidate``. + + ``update(host)`` writes the leading ``host.numel()`` elements, and a recorded longer run whose prefix matches still + holds them, so it is skipped. ``update(host, view=v)`` writes ``host`` into the view ``v`` of the buffer's storage + instead and is skipped only for the same view geometry and host shape. The host values are snapshotted when the + copy runs eagerly (into pinned memory with ``pin_memory``), since owners reuse their host buffers; a merged copy is + snapshotted into the pack instead. ``snapshot=False`` skips the eager snapshot for an owner that passes a host + tensor it does not write again. + + ``fill_tail`` fills ``tensor[start:]`` and records the fill (start, value), so an identical fill is skipped. Call it + after every write of the live rows ``tensor[:start]``, including one that covers the whole buffer (``start == + len(tensor)``, which fills nothing): a live-row write past the recorded start overwrites the tail. + """ + + def __init__( + self, + tensor: torch.Tensor, + *, + site: Optional[CopySiteEnum], + pin_memory: bool = False, + snapshot: bool = True, + ) -> None: + if site is not None and site.direction is not Direction.H2D: + raise ValueError( + f"DeviceCopyBuffer uploads from the host; {_site_name(site)} is not an h2d site" + ) + self._tensor = tensor + self._site = site + self._pin_memory = pin_memory + self._snapshot = self._stage if snapshot else None + self._last_host_values = None + self._last_host_shape: torch.Size | None = None + self._copy_key: object = None + # (view, start, value) of the last tail fill. + self._tail: Optional[tuple] = None + + @property + def tensor(self) -> torch.Tensor: + return self._tensor + + @property + def site(self) -> Optional[CopySiteEnum]: + return self._site + + @property + def host_values(self) -> Optional[torch.Tensor]: + """Last recorded host values, reshaped to the submitted view; callers must not mutate them.""" + if self._last_host_values is None: + return None + count = self._last_host_shape.numel() + return self._last_host_values[:count].view(self._last_host_shape) + + def invalidate(self) -> None: + """Forget a pending or externally overwritten copy (and the tail fill).""" + self._last_host_values = None + self._last_host_shape = None + self._copy_key = None + self._tail = None + + def matches(self, host: torch.Tensor, *, key: object = None, exact: bool = False) -> bool: + """Whether the recorded host inputs match, including any copy-specific layout key. + + ``exact`` requires the same host shape instead of accepting a matching prefix. + Callers that transform or defer their copies must record only successful submissions + and invalidate the record if the submission is discarded or another writer intervenes. + """ + last = self._last_host_values + flat = host.reshape(-1) + return ( + last is not None + and key == self._copy_key + and last.dtype == host.dtype + and (not exact or self._last_host_shape == host.shape) + and last.numel() >= flat.numel() + and torch.equal(last[: flat.numel()], flat) + ) + + def record(self, host: torch.Tensor, *, key: object = None) -> None: + """Remember the inputs of a copy issued by this buffer or a specialized copy path.""" + flat = host.reshape(-1) + last = self._last_host_values + if ( + last is None + or key != self._copy_key + or last.dtype != host.dtype + or last.numel() <= flat.numel() + ): + self._last_host_values = flat.clone() + else: + last = last.clone() + last[: flat.numel()] = flat + self._last_host_values = last + self._last_host_shape = host.shape + self._copy_key = key + + def _stage(self, src: torch.Tensor) -> torch.Tensor: + staging = torch.empty_like(src, device="cpu", pin_memory=self._pin_memory) + staging.copy_(src) + return staging + + # A changed-span upload is used only while it moves at most this fraction of the view's elements. + CHANGED_SPAN_MAX_FRACTION = 0.5 + + def update( + self, + host: torch.Tensor, + force: bool = False, + *, + view: Optional[torch.Tensor] = None, + changed_span: bool = False, + ) -> bool: + """Upload ``host`` through the buffer's site unless the record says the buffer holds it; True if issued. + + ``force`` issues the copy whatever the record says. ``view``: the destination, a view of the buffer's storage + (default: the buffer's leading ``host.numel()`` elements). + + ``changed_span``: a changed upload into a 2-D ``view`` with contiguous rows, merged into the open pack through a + site in the plan's ``changed_span``, whose record covers the same rows (same first element, row count and row + stride) issues, per row, only the span from its first to its last changed column, plus any columns the record + does not cover, each through the buffer's site. A page table that gained one page then stages a few bytes + instead of every padded row. It falls back to the whole view when the spans would move more than + ``CHANGED_SPAN_MAX_FRACTION`` of it, or when the copy is not merged (per-row eager copies would cost more than + one). + """ + site = self._site + if site is None: + raise ValueError("a record-only DeviceCopyBuffer (site=None) does not issue copies") + if view is None: + src = host.reshape(-1) + dst = self._tensor.view(-1)[: src.numel()] + key, exact = None, False + if self._tail is not None and src.numel() > self._tail[1]: + self._tail = None + else: + src, dst = host, view + key, exact = _view_key(view), True + self._tail = None + if _capturing(dst): + self.invalidate() + return copy(site, dst, src, snapshot=self._snapshot) + if not force and self.matches(host, key=key, exact=exact): + return copy(site, dst, src, unchanged=True) + # Dropped while the copy is issued, then recorded on top of what the buffer held (a longer recorded run + # keeps its tail). + previous = (self._last_host_values, self._last_host_shape, self._copy_key) + old = ( + (self._copy_key, self.host_values) + if changed_span and view is not None and self._last_host_values is not None + else None + ) + self._last_host_values = self._last_host_shape = self._copy_key = None + if old is not None and self._copy_changed_span(site, dst, src, *old): + issued = True + else: + issued = copy(site, dst, src, on_drop=self.invalidate, snapshot=self._snapshot) + self._last_host_values, self._last_host_shape, self._copy_key = previous + self.record(host, key=key) + return issued + + def _copy_changed_span( + self, + site: CopySiteEnum, + dst: torch.Tensor, + src: torch.Tensor, + old_key: object, + old: torch.Tensor, + ) -> bool: + """Stage only the changed span of each row of ``dst`` (see ``update``); False if not applicable.""" + if not isinstance(old_key, tuple) or not stages_changed_span(site): + return False + offset, shape, stride, dtype = old_key + if ( + dst.dim() != 2 + or dst.stride(1) != 1 + or offset != dst.storage_offset() + or len(shape) != 2 + or shape[0] != dst.shape[0] + or stride != tuple(dst.stride()) + or dtype != dst.dtype + or old.dtype != src.dtype + or tuple(src.shape) != tuple(dst.shape) + ): + return False + cols = dst.shape[1] + known = min(shape[1], cols) + changed = src[:, :known] != old[:, :known] + if cols > known: + # Columns the record does not cover hold unknown device values. + changed = torch.cat([changed, changed.new_ones(dst.shape[0], cols - known)], dim=1) + rows = changed.any(dim=1).nonzero().flatten().tolist() + spans = [] + for row in rows: + columns = changed[row].nonzero().flatten() + spans.append((row, int(columns[0]), int(columns[-1]) + 1)) + if sum(hi - lo for _, lo, hi in spans) > self.CHANGED_SPAN_MAX_FRACTION * dst.numel(): + return False + for row, lo, hi in spans: + copy(site, dst[row, lo:hi], src[row, lo:hi], on_drop=self.invalidate) + return True + + def fill_tail(self, site: CopySiteEnum, start: int, value: int) -> None: + """``fill_tail(site, tensor, start, value)``, passing unchanged=True when the last fill was the same.""" + buf = self._tensor + if start >= buf.shape[0] or _capturing(buf): + self._tail = None + fill_tail(site, buf, start, value) + return + key = (_view_key(buf), start, value) + unchanged = self._tail == key + if not unchanged: + self._tail = None + fill_tail(site, buf, start, value, unchanged=unchanged) + self._tail = key diff --git a/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/inputs.py b/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/inputs.py index 1aae209a3759..3f2abc9beb4d 100644 --- a/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/inputs.py +++ b/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/inputs.py @@ -26,13 +26,14 @@ from tensorrt_llm._torch.kernels.spec_step_copies import op as spec_step_copies from tensorrt_llm._torch.metadata import KVCacheParams from tensorrt_llm._torch.models.modeling_multimodal_mixin import _build_request_multimodal_input -from tensorrt_llm._torch.pyexecutor import step_inputs +from tensorrt_llm._torch.pyexecutor import device_copy, step_inputs from tensorrt_llm._torch.pyexecutor.breakable_cuda_graph import enable_breakable_cuda_graph from tensorrt_llm._torch.pyexecutor.breakable_cuda_graph_runner import BreakableCUDAGraphRunner from tensorrt_llm._torch.pyexecutor.cuda_graph_runner import ( CUDAGraphRunner, get_mrope_dummy_seq_slot, ) +from tensorrt_llm._torch.pyexecutor.device_copy import CopySite from tensorrt_llm._torch.pyexecutor.guided_decoder import CapturableGuidedDecoder from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import KVCacheManagerV2 from tensorrt_llm._torch.pyexecutor.llm_request import ( @@ -2500,9 +2501,11 @@ def previous_pos_indices_device(tokens_per_request: Sequence[int]) -> int: assert total_num_tokens <= self._config.max_num_tokens, ( f"total_num_tokens ({total_num_tokens}) should be less than or equal to max_num_tokens ({self._config.max_num_tokens})" # noqa: E501 ) + # An open copy-plan pack owns the step's copies: keep the per-buffer copies then. staged = ( self._spec_overlap_staging and enable_spec_decode + and not device_copy.is_open() and next_draft_tokens_device is not None and previous_batch_len > 0 and self._stage_spec_overlap_inputs( @@ -2809,7 +2812,11 @@ def _copy_position_ids_to_device( generation_positions.view(1, 1, -1).expand(3, 1, -1), ) else: - step_inputs.copy_(self._buffers.position_ids_cuda[:total_num_tokens], host_position_ids) + position_ids_cuda = self._buffers.position_ids_cuda[:total_num_tokens] + if device_copy.current() is not None: + device_copy.copy(CopySite.INPUT_POSITION_IDS, position_ids_cuda, host_position_ids) + else: + step_inputs.copy_(position_ids_cuda, host_position_ids) host.host_position_ids = host_position_ids diff --git a/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py b/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py index 856dc2d6fb77..ef235b8a1b95 100644 --- a/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py +++ b/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py @@ -31,7 +31,7 @@ from tensorrt_llm._torch.moe.workspace import CutlassWorkspaceReclaimer from tensorrt_llm._torch.peft.lora.config import LoraConfig from tensorrt_llm._torch.peft.lora.manager import LoraModelConfig -from tensorrt_llm._torch.pyexecutor import h2d_cache, step_inputs +from tensorrt_llm._torch.pyexecutor import device_copy, h2d_cache, step_inputs from tensorrt_llm._torch.pyexecutor.breakable_cuda_graph_runner import BreakableCUDAGraphRunner from tensorrt_llm._torch.pyexecutor.context_logits_plan import ContextLogitsPlan from tensorrt_llm._torch.pyexecutor.cuda_graph_runner import ( @@ -40,6 +40,7 @@ DSparkCUDAGraphRunner, KeyType, ) +from tensorrt_llm._torch.pyexecutor.device_copy import CopyPlan, DeviceCopier, copy_plan_of from tensorrt_llm._torch.pyexecutor.guided_decoder import CapturableGuidedDecoder from tensorrt_llm._torch.pyexecutor.h2d_cache import H2DSkipCache from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import KVCacheManagerV2 @@ -157,6 +158,19 @@ def __getattr__(self, name: str) -> Any: return getattr(super().__getattr__("eager_model"), name) +def check_one_copy_path(model: object, plan: CopyPlan, hooks: RuntimeHooks) -> None: + """Refuse a model that declares a copy plan and also turns on the step-input pack or the skip cache. + + The two paths coexist across models, not within one: a model moves its step's copies to the copy plan as a whole. + """ + if plan and (hooks.pack_step_inputs or hooks.skip_unchanged_h2d): + raise ValueError( + f"{type(model).__name__} declares a copy plan and also turns on the step-input pack or the skip cache " + "(RuntimeHooks.pack_step_inputs / skip_unchanged_h2d); a model moves its step's copies to the copy plan " + "as a whole" + ) + + class DecoderRunner(ScheduledModelRunner): """Run scheduled decoder generation with its own buffers, graphs and metadata.""" @@ -165,6 +179,10 @@ class DecoderRunner(ScheduledModelRunner): # This engine's skip cache and step-input pack (None: the hook is off), active only during its own forward. _h2d_cache: H2DSkipCache | None = None _step_pack: StepInputPack | None = None + # The copy plan the model declares (device_copy); empty by default, and then the engine has no copier. + copy_plan: CopyPlan = CopyPlan() + # This engine's copier (None: an empty plan), active only during its own forward. + _copier: DeviceCopier | None = None # Whether a batch may keep the greedy tokens a forward sampled in place of the logits. False when the executor # runs a guided decoder, which reads the logits of every batch. keep_greedy_only_outputs = True @@ -250,6 +268,26 @@ def __init__( ) if self._h2d_cache is not None and self._h2d_cache.changed_span: logger.info("[M3 host] changed metadata uploads stage only each row's changed span") + self.copy_plan = copy_plan_of(model) + check_one_copy_path(model, self.copy_plan, self.runtime_hooks) + self._copier = ( + DeviceCopier(self.copy_plan, torch.device("cuda", torch.cuda.current_device())) + if self.copy_plan + else None + ) + if self._copier is not None: + final = self.copy_plan.final_flush + logger.info( + f"[device copy] {len(self.copy_plan.merge)} merged sites, " + f"{len(self.copy_plan.flush_after)} barriers, " + f"parallel scatter {self.copy_plan.parallel_scatter}, final flush " + + ( + "eager" + if final is None + else f"in graph (early copy {final.early_copy}, " + f"one-launch prologue {final.prologue_op is not None}, step epoch {final.step_epoch})" + ) + ) # Heads that can project context logits eagerly; the model is fixed. self._context_logits_heads = context_logits_heads( model, mapping, is_spec_decode=config.is_spec_decode @@ -430,7 +468,11 @@ def forward( resource_manager: ResourceManager, is_dummy: bool = False, ) -> Any: - with h2d_cache.active(self._h2d_cache), step_inputs.active(self._step_pack): + with ( + h2d_cache.active(self._h2d_cache), + step_inputs.active(self._step_pack), + device_copy.active(self._copier), + ): try: return self._forward_decoder(inputs, resource_manager, is_dummy=is_dummy) except BaseException: @@ -438,6 +480,9 @@ def forward( step_inputs.discard() if self._h2d_cache is not None: self._h2d_cache.clear() + # The copy plan's pack likewise: discarding it runs the staged copies' on_drop callbacks, so the + # buffer owners forget the records they made for them. + device_copy.discard() raise def _initialize_moe_workspace_reclaimer(self) -> None: @@ -819,6 +864,15 @@ def _forward_decoder( and not self.cuda_graph_runner.needs_capture(key) ): step_inputs.begin() + # The copy plan's pack, under the same condition, for a model that declares a plan. + if ( + self._copier is not None + and self._copier.merges + and can_run_graph + and not self.cuda_graph_runner.is_warmup_only + and not self.cuda_graph_runner.needs_capture(key) + ): + device_copy.begin() with self._cuda_graph_metadata_prepare_scope( attn_metadata, key, @@ -910,6 +964,7 @@ def postprocess_inputs(model_inputs: dict[str, Any]) -> None: ) # No-op unless a pack was opened above and the replay did not issue it. step_inputs.flush() + device_copy.flush() if context_plan is not None and context_plan.handled: outputs = dict(outputs) if isinstance(outputs, dict) else {"logits": outputs} outputs["context_logits_handled"] = True diff --git a/tensorrt_llm/_torch/speculative/interface.py b/tensorrt_llm/_torch/speculative/interface.py index 3828dcdb1f4f..3acfac14291d 100644 --- a/tensorrt_llm/_torch/speculative/interface.py +++ b/tensorrt_llm/_torch/speculative/interface.py @@ -38,7 +38,8 @@ from ..flashinfer_utils import IS_FLASHINFER_AVAILABLE from ..modules.bf16_draft_head import BF16DraftHead, build_bf16_draft_head from ..modules.logits_processor import LogitsProcessor -from ..pyexecutor import step_inputs +from ..pyexecutor import device_copy, step_inputs +from ..pyexecutor.device_copy import CopySite from ..pyexecutor.resource_manager import ResourceManagerType if TYPE_CHECKING: @@ -699,11 +700,17 @@ def one_model_sampling_mode(requests, spec_metadata: "SpecMetadata") -> int: def _upload_host_values(dst: torch.Tensor, values: np.ndarray, - pack: bool = False) -> None: - """Copy ``values`` into the head of ``dst`` without blocking; ``pack`` lets the copy move to the open - step-input pack's flush (a plain copy when no pack is open).""" + pack: bool = False, + site: Optional[CopySite] = None) -> None: + """Copy ``values`` into the head of ``dst`` without blocking. + + With ``site``, a model that declares a copy plan (an active copier) may merge the copy. Otherwise ``pack`` lets the + copy move to the open step-input pack's flush (a plain copy when no pack is open). + """ src = maybe_pin_memory(torch.from_numpy(values)) - if pack: + if site is not None and device_copy.current() is not None: + device_copy.copy(site, dst[:len(values)], src) + elif pack: step_inputs.copy_(dst[:len(values)], src) else: dst[:len(values)].copy_(src, non_blocking=True) @@ -1213,10 +1220,14 @@ def _populate_request_rng_state( # The forward is the first reader of these buffers. pack = self.pack_rng_uploads - _upload_host_values(self.request_seeds, request_seeds, pack) - _upload_host_values(self.request_offsets, request_offsets, pack) - _upload_host_values(self.seeds, flat_seeds, pack) - _upload_host_values(self.offsets, flat_offsets, pack) + _upload_host_values(self.request_seeds, request_seeds, pack, + CopySite.SPEC_RNG_REQUEST_SEEDS) + _upload_host_values(self.request_offsets, request_offsets, pack, + CopySite.SPEC_RNG_REQUEST_OFFSETS) + _upload_host_values(self.seeds, flat_seeds, pack, + CopySite.SPEC_RNG_SEEDS) + _upload_host_values(self.offsets, flat_offsets, pack, + CopySite.SPEC_RNG_OFFSETS) def prepare_rejection_sampling_buffers(self): """ diff --git a/tests/unittest/_torch/pyexecutor/test_context_logits_memory.py b/tests/unittest/_torch/pyexecutor/test_context_logits_memory.py index 0625465c05c0..f43bb7c54fab 100644 --- a/tests/unittest/_torch/pyexecutor/test_context_logits_memory.py +++ b/tests/unittest/_torch/pyexecutor/test_context_logits_memory.py @@ -357,6 +357,7 @@ def run(graph_key, inputs, run_model, postprocess_inputs, **kwargs): _model=model, _mapping=mapping, runtime_hooks=SimpleNamespace(pack_step_inputs=False), + _copier=None, _context_logits_heads=context_logits_heads(model, mapping, is_spec_decode=False), _config=SimpleNamespace(is_spec_decode=False, spec_config=None, use_mrope=False), _graphs=graphs, diff --git a/tests/unittest/_torch/pyexecutor/test_cuda_graph_runner.py b/tests/unittest/_torch/pyexecutor/test_cuda_graph_runner.py index e1c6acbcfbeb..67f7bfb83710 100644 --- a/tests/unittest/_torch/pyexecutor/test_cuda_graph_runner.py +++ b/tests/unittest/_torch/pyexecutor/test_cuda_graph_runner.py @@ -52,6 +52,7 @@ def _empty_runner() -> CUDAGraphRunner: runner.graph_outputs = {} runner.graph_metadata = {} runner._step_input_prologue_keys = {} + runner._prologue_copiers = {} runner._hostfunc_graph_keys = set() runner.forward_scratch_buffers = Buffers() return runner @@ -729,6 +730,7 @@ def test_replay_defers_copies_only_to_its_captured_pack(monkeypatch, same_pack, runner._check_strict_buffer_stability = lambda meta: None runner._get_num_tokens_for_key = lambda key: 1 runner._step_input_prologue_keys = {key: captured} + runner._prologue_copiers = {} runner._pre_replay_hooks = [] runner._hostfunc_graph_keys = set() @@ -767,6 +769,7 @@ def test_forward_scratch_is_released_with_runner(monkeypatch: pytest.MonkeyPatch runner.graphs = {} runner.graph_outputs = {} runner._step_input_prologue_keys = {} + runner._prologue_copiers = {} runner._hostfunc_graph_keys = set() runner.forward_scratch_buffers = Buffers() scratch = torch.empty(16) diff --git a/tests/unittest/_torch/pyexecutor/test_device_copy.py b/tests/unittest/_torch/pyexecutor/test_device_copy.py new file mode 100644 index 000000000000..d52cfbed62c3 --- /dev/null +++ b/tests/unittest/_torch/pyexecutor/test_device_copy.py @@ -0,0 +1,1288 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""A step's copies routed by a copy plan: merged into one upload and one scatter launch per barrier. Whether a copy is +needed is the buffer owner's call (``unchanged``, ``DeviceCopyBuffer``).""" + +import threading + +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor import device_copy +from tensorrt_llm._torch.pyexecutor.device_copy import ( + CHECK_ENV, + CopyPlan, + CopySite, + CopySiteEnum, + DeviceCopier, + Direction, + _Pack, + copy_plan_of, +) + + +def _scatter_op_registered() -> bool: + return torch.cuda.is_available() and hasattr(torch.ops.trtllm, "step_inputs_scatter") + + +pytestmark = [ + pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required"), + pytest.mark.skipif( + not _scatter_op_registered(), + reason="torch.ops.trtllm.step_inputs_scatter is not registered (built from " + "cpp/tensorrt_llm/thop/minimaxM3/stepInputsScatterOp.cpp)", + ), +] + +# About half a second of GPU time: long enough that work queued behind it is still pending when +# the host checks, short enough for a unit test. +_SLEEP_CYCLES = 1_000_000_000 + + +class _Site(CopySiteEnum): + H2D = ("test.h2d", Direction.H2D) + D2D = ("test.d2d", Direction.D2D) + # Merged; its owner skips unchanged uploads (_REC). + SKIP = ("test.skip", Direction.H2D) + TAIL = ("test.tail", Direction.FILL) + # Not in the plan. + PLAIN = ("test.plain", Direction.H2D) + # Not in the plan; its owner skips unchanged uploads. + SKIP_ONLY = ("test.skip_only", Direction.H2D) + + +_PLAN = CopyPlan(merge={_Site.H2D, _Site.D2D, _Site.SKIP}) + + +class _Owner: + """A buffer owner keeping a DeviceCopyBuffer per destination storage (the owner pattern of a metadata class that + skips unchanged uploads), replaced when written through another site.""" + + def __init__(self) -> None: + self._buffers: dict[int, device_copy.DeviceCopyBuffer] = {} + + def buffer(self, site, t: torch.Tensor) -> device_copy.DeviceCopyBuffer: + addr = t.untyped_storage().data_ptr() + buffer = self._buffers.get(addr) + if buffer is None or buffer.site is not site: + buffer = device_copy.DeviceCopyBuffer(t, site=site, snapshot=False) + self._buffers[addr] = buffer + return buffer + + def copy(self, site, dst: torch.Tensor, src: torch.Tensor) -> bool: + return self.buffer(site, dst).update(src, view=dst) + + def fill_tail(self, site, buf: torch.Tensor, start: int, value: int) -> None: + self.buffer(None, buf).fill_tail(site, start, value) + + def forget(self, t: torch.Tensor) -> None: + """Another writer wrote ``t``: drop its record.""" + buffer = self._buffers.get(t.untyped_storage().data_ptr()) + if buffer is not None: + buffer.invalidate() + + +# The test's buffer owner: what its skippable uploads last wrote (fresh for every test). +_REC = _Owner() + + +@pytest.fixture(autouse=True) +def _fresh_records(monkeypatch): + global _REC + _REC = _Owner() + # The tests here turn check mode on where they test it (the unit-test conftest turns it on everywhere). + monkeypatch.setattr(device_copy, "CHECK_MODE", False) + yield + + +def _device(): + return torch.device("cuda", torch.cuda.current_device()) + + +def _new_copier(plan: CopyPlan = _PLAN) -> DeviceCopier: + return DeviceCopier(plan, _device()) + + +@pytest.fixture +def copier(): + """A fresh copier, active for the test body.""" + active = _new_copier() + with device_copy.active(active): + yield active + device_copy.flush() + torch.cuda.synchronize() + + +def _copy(dst, src) -> bool: + """A copy through the merged site of ``src``'s direction.""" + return device_copy.copy(_Site.H2D if src.device.type == "cpu" else _Site.D2D, dst, src) + + +def _skip_copy(dst, src) -> bool: + """A merged copy whose owner skips it when its record says ``dst`` already holds ``src``.""" + return _REC.copy(_Site.SKIP, dst, src) + + +def _host(values, dtype=torch.int32): + return torch.tensor(values, dtype=dtype).pin_memory() + + +def _zeros(*shape, dtype=torch.int32): + return torch.zeros(*shape, dtype=dtype, device="cuda") + + +# --------------------------------------------------------------------------- +# The plan +# --------------------------------------------------------------------------- + + +def test_the_plan_routes_each_site(): + plan = CopyPlan(merge={_Site.H2D}) + merged, plain = _zeros(2), _zeros(2) + values = _host([4, 5]) + + with device_copy.active(_new_copier(plan)): + device_copy.begin() + # A site not in the plan is a plain copy, every time. + assert [device_copy.copy(_Site.PLAIN, plain, values) for _ in range(2)] == [True, True] + assert device_copy.copy(_Site.H2D, merged, values) is True + # The caller's unchanged=True is not issued, merged or not. + assert device_copy.copy(_Site.PLAIN, plain, values, unchanged=True) is False + assert device_copy.copy(_Site.H2D, merged, _host([9, 9]), unchanged=True) is False + torch.cuda.synchronize() + assert (merged.tolist(), plain.tolist()) == ([0, 0], [4, 5]) + with pytest.raises(ValueError, match=r"_Site\.H2D is declared h2d"): + device_copy.copy(_Site.H2D, merged, torch.ones(2, dtype=torch.int32, device="cuda")) + device_copy.flush() + torch.cuda.synchronize() + assert merged.tolist() == [4, 5] + + with pytest.raises(ValueError, match="cannot be merged"): + CopyPlan(merge={_Site.TAIL}) + with pytest.raises(TypeError, match="CopySiteEnum"): + CopyPlan(merge={"attn.kv_lens"}) + model = torch.nn.Module() + assert copy_plan_of(model) == CopyPlan() and not copy_plan_of(model) + model.copy_plan = plan + assert copy_plan_of(model) is plan + model.copy_plan = {_Site.H2D} + with pytest.raises(TypeError, match=r"Module\.copy_plan must be a CopyPlan"): + copy_plan_of(model) + + +@pytest.mark.parametrize( + "broken", ["merged_dst_written", "d2d_src_changed", "skipped_dst_written", "tail_written"] +) +def test_check_mode_names_the_site_whose_assertion_broke(monkeypatch, broken): + monkeypatch.setattr(device_copy, "CHECK_MODE", True) + checked = _new_copier() + dst = _zeros(4) + src = torch.arange(4, dtype=torch.int32, device="cuda") + + with device_copy.active(checked): + # The legitimate pattern passes: a device copy reading what a host copy of the same pack writes. + staged, static = _zeros(4), _zeros(4) + device_copy.begin() + _copy(staged, _host([1, 2, 3, 4])) + _copy(static, staged) + device_copy.flush() + torch.cuda.synchronize() + assert static.tolist() == [1, 2, 3, 4] + + if broken == "merged_dst_written": + device_copy.begin() + _copy(dst, _host([1, 1, 1, 1])) + dst[0] = 7 + site, call = r"_Site\.H2D", device_copy.flush + elif broken == "d2d_src_changed": + device_copy.begin() + _copy(dst, src) + src.add_(1) + site, call = r"_Site\.D2D", device_copy.flush + elif broken == "skipped_dst_written": + _REC.copy(_Site.SKIP_ONLY, dst, _host([3, 3, 3, 3])) + dst[1] = 7 + site, call = ( + r"_Site\.SKIP_ONLY", + lambda: _REC.copy(_Site.SKIP_ONLY, dst, _host([3, 3, 3, 3])), + ) + else: + _REC.fill_tail(_Site.TAIL, dst, 1, -1) + dst[2] = 7 + site, call = r"_Site\.TAIL", lambda: _REC.fill_tail(_Site.TAIL, dst, 1, -1) + with pytest.raises(RuntimeError, match=f"{CHECK_ENV}: .*{site}"): + call() + device_copy.discard() + + +# --------------------------------------------------------------------------- +# No active copier +# --------------------------------------------------------------------------- + + +def test_without_an_active_copier_every_call_is_a_plain_copy(): + dst = _zeros(4) + d2d_dst = _zeros(4) + src_dev = torch.arange(4, dtype=torch.int32, device="cuda") + tail = _zeros(6) + + device_copy.begin() + assert not device_copy.is_open() + for _ in range(2): + assert _copy(dst, _host([1, 2, 3, 4])) is True + _copy(d2d_dst, src_dev) + device_copy.fill_tail(_Site.TAIL, tail, 2, -1) + tail[3] = 5 + device_copy.fill_tail(_Site.TAIL, tail, 2, -1) + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3, 4] + assert d2d_dst.tolist() == [0, 1, 2, 3] + assert tail.tolist() == [0, 0, -1, -1, -1, -1] + # The owner's unchanged=True is honored without a copier too. + assert device_copy.copy(_Site.SKIP, dst, _host([9, 9, 9, 9]), unchanged=True) is False + assert _skip_copy(dst, _host([1, 2, 3, 4])) is True + assert _skip_copy(dst, _host([1, 2, 3, 4])) is False + device_copy.flush() + device_copy.discard() + + +def test_an_inactive_copier_is_not_used(): + idle = _new_copier() + dst = _zeros(2) + + _copy(dst, _host([3, 4])) + torch.cuda.synchronize() + + assert dst.tolist() == [3, 4] + assert not idle.is_open and idle._pack.staged == [] + + +# --------------------------------------------------------------------------- +# Merging and ordering +# --------------------------------------------------------------------------- + + +def test_merged_host_copy_lands_only_at_flush(copier): + dst = _zeros(4) + src = _host([1, 2, 3, 4]) + + device_copy.begin() + assert device_copy.is_open() and copier.is_open + assert _copy(dst, src) is True + # The pack snapshots the host bytes at the copy site. + src.fill_(7) + torch.cuda.synchronize() + assert dst.tolist() == [0, 0, 0, 0] + + device_copy.flush() + assert not device_copy.is_open() + assert copier._pack.staged == [] + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3, 4] + + +def test_merged_host_copy_into_a_row_strided_2d_slice(copier): + table = torch.full((4, 6), -1, dtype=torch.int32, device="cuda") + dst = table[1:3, 2:5] + assert not dst.is_contiguous() and dst.stride(1) == 1 + expected = table.clone() + expected[1:3, 2:5] = torch.tensor([[1, 2, 3], [4, 5, 6]], dtype=torch.int32) + + device_copy.begin() + _copy(dst, _host([[1, 2, 3], [4, 5, 6]])) + torch.cuda.synchronize() + assert torch.equal(table, torch.full_like(table, -1)) + + device_copy.flush() + torch.cuda.synchronize() + assert torch.equal(table, expected) + + +def test_merged_host_copy_converts_dtype_on_the_host(copier): + dst = _zeros(3) + + device_copy.begin() + _copy(dst, _host([7, -8, 9], dtype=torch.int64)) + device_copy.flush() + torch.cuda.synchronize() + + assert dst.tolist() == [7, -8, 9] + + +def test_merged_device_copy_lands_only_at_flush(copier): + dst = _zeros(5) + src = torch.arange(5, dtype=torch.int32, device="cuda") + + device_copy.begin() + _copy(dst, src) + torch.cuda.synchronize() + assert dst.tolist() == [0] * 5 + + device_copy.flush() + torch.cuda.synchronize() + assert dst.tolist() == [0, 1, 2, 3, 4] + + +def test_device_copy_reads_what_a_host_copy_of_the_same_pack_wrote(copier): + """The graph runner copies the engine's freshly uploaded buffers into its static inputs.""" + staged = _zeros(4) + static = _zeros(1, 8) + + device_copy.begin() + _copy(staged, _host([10, 11, 12, 13])) + _copy(static[:, :4], staged.unsqueeze(0)) + device_copy.flush() + torch.cuda.synchronize() + + assert static.tolist() == [[10, 11, 12, 13, 0, 0, 0, 0]] + + +# The scatter's CTA copies descriptor i with thread t on bytes t, t + 256, ... of it. When the +# second copy starts TAIL bytes before the end of the first (TAIL not a multiple of the warp +# size), its first bytes are the ones the first copy writes last, from other warps: without a +# barrier between descriptors, a warp that finished the first copy early writes (or reads) them +# before a slower warp has written its part of the first copy. +_TAILS = [239, 1000, 4073] + + +@pytest.mark.parametrize("tail", _TAILS) +@pytest.mark.parametrize("kind", ["host", "device", "device_then_host"]) +def test_overlapping_destinations_land_in_call_order(copier, kind, tail): + """The later copy wins on the overlap, as it would eagerly. + + ``device_then_host``: the scatter writes host copies before device copies, so a host copy into bytes a staged + device copy writes flushes the pack first. + """ + # Host copies share the 64 KiB staging buffer; device copies are bounded only by memory, and a + # long first copy lets the warps drift apart. + n = 24 * 1024 if kind == "host" else 1 << 20 + m = 8 * 1024 + first = torch.full((n,), 1, dtype=torch.uint8) + second = torch.full((m,), 2, dtype=torch.uint8) + first = first.pin_memory() if kind == "host" else first.cuda() + second = second.cuda() if kind == "device" else second.pin_memory() + start = n - tail + expected = torch.ones(start + m, dtype=torch.uint8) + expected[start:] = 2 + + mismatches = [] + for trial in range(50): + buf = torch.zeros(start + m, dtype=torch.uint8, device="cuda") + device_copy.begin() + _copy(buf[:n], first) + _copy(buf[start:], second) + device_copy.flush() + torch.cuda.synchronize() + wrong = int((buf.cpu() != expected).sum()) + if wrong: + mismatches.append((trial, wrong)) + + assert not mismatches, f"(trial, overlap bytes where the earlier copy won): {mismatches}" + + +@pytest.mark.parametrize("tail", _TAILS) +def test_a_device_copy_reads_the_destination_of_an_earlier_device_copy(copier, tail): + """A chain of device copies: the second reads the bytes the first writes last.""" + n = 1 << 20 + source = torch.randint(1, 256, (n,), dtype=torch.uint8, device="cuda") + expected = source[n - tail :].clone() + + mismatches = [] + for trial in range(50): + middle = torch.zeros(n, dtype=torch.uint8, device="cuda") + out = torch.zeros(tail, dtype=torch.uint8, device="cuda") + device_copy.begin() + _copy(middle, source) + _copy(out, middle[n - tail :]) + device_copy.flush() + torch.cuda.synchronize() + wrong = int((out != expected).sum()) + if wrong: + mismatches.append((trial, wrong)) + + assert not mismatches, ( + f"(trial, stale bytes read before the first copy wrote them): {mismatches}" + ) + + +# --------------------------------------------------------------------------- +# Staged tensors stay alive +# --------------------------------------------------------------------------- + + +def test_a_temporary_device_source_outlives_its_copy(copier): + """The pack holds the source until the scatter is queued, so its memory is not reused.""" + numel = 1 << 16 + expected = torch.arange(numel, dtype=torch.int32) * 3 + 1 + dst = _zeros(numel) + + device_copy.begin() + # The only reference to this source is the one the pack keeps. + _copy(dst, torch.arange(numel, dtype=torch.int32, device="cuda") * 3 + 1) + assert len(copier._pack.staged) == 2 + # Churn the allocator with same-sized blocks: a freed source block would be handed out here. + garbage = [torch.full((numel,), -7, dtype=torch.int32, device="cuda") for _ in range(16)] + torch.cuda.synchronize() + device_copy.flush() + del garbage + # Blocks allocated after the flush are ordered behind the scatter on this stream. + after = [torch.full((numel,), -9, dtype=torch.int32, device="cuda") for _ in range(16)] + torch.cuda.synchronize() + + assert torch.equal(dst.cpu(), expected) + del after + + +def test_a_temporary_destination_view_outlives_its_copy(copier): + owner = _zeros(8) + + device_copy.begin() + _copy(owner.view(2, 4)[1], _host([5, 6, 7, 8])) + assert len(copier._pack.staged) == 1 + device_copy.flush() + torch.cuda.synchronize() + + assert owner.tolist() == [0, 0, 0, 0, 5, 6, 7, 8] + + +# --------------------------------------------------------------------------- +# Fallbacks +# --------------------------------------------------------------------------- + + +def test_an_unpackable_copy_flushes_first_and_keeps_order(copier): + buf = _zeros(8) + other = _zeros(4) + later = _zeros(2) + + device_copy.begin() + _copy(buf, _host([1] * 8)) + _copy(other, _host([3, 3, 3, 3])) + # A strided 1-D destination cannot be packed: the pack lands first, then this copy. + _copy(buf[::2], _host([2, 2, 2, 2])) + assert device_copy.is_open() + torch.cuda.synchronize() + assert buf.tolist() == [2, 1, 2, 1, 2, 1, 2, 1] + assert other.tolist() == [3, 3, 3, 3] + + # The pack stays open: later copies are merged again. + _copy(later, _host([5, 6])) + torch.cuda.synchronize() + assert later.tolist() == [0, 0] + device_copy.flush() + torch.cuda.synchronize() + assert later.tolist() == [5, 6] + + +def test_an_eager_consumer_of_a_merged_copy_flushes_its_producer_first(): + # A partial plan: the decoder's position ids are merged, the graph runner's copy of them into the static input is + # not. That eager copy reads the merged copy's destination, so the producer lands first. + copier = _new_copier(CopyPlan(merge={CopySite.INPUT_POSITION_IDS})) + positions, static = _zeros(4), _zeros(4) + with device_copy.active(copier): + device_copy.begin() + device_copy.copy(CopySite.INPUT_POSITION_IDS, positions, _host([7, 8, 9, 10])) + device_copy.copy(CopySite.GRAPH_POSITION_IDS, static, positions) + device_copy.flush() + torch.cuda.synchronize() + assert static.tolist() == positions.tolist() == [7, 8, 9, 10] + + +@pytest.mark.parametrize("eager", ["host_copy", "fill"]) +def test_an_eager_write_issues_the_staged_copies_first(copier, eager): + buf = _zeros(4) + pack = copier._pack + + device_copy.begin() + _copy(buf, _host([1, 2, 3, 4])) + if eager == "host_copy": + device_copy.copy(_Site.PLAIN, buf[2:], _host([0, 0])) + else: + device_copy.fill_tail(_Site.TAIL, buf, 2, 0) + # The staged copy was issued before the eager write, and the pack stays open. + assert pack.seg > 0 and device_copy.is_open() + device_copy.flush() + torch.cuda.synchronize() + assert buf.tolist() == [1, 2, 0, 0] + + +def test_a_device_copy_with_a_dtype_change_is_not_merged(copier): + dst = _zeros(3, dtype=torch.int64) + src = torch.tensor([4, 5, 6], dtype=torch.int32, device="cuda") + + device_copy.begin() + _copy(dst, src) + torch.cuda.synchronize() + + assert dst.tolist() == [4, 5, 6] + assert device_copy.is_open() + + +def test_a_copy_larger_than_the_staging_buffer_falls_back_in_order(copier): + numel = _Pack.CAPACITY // 4 + big = _zeros(numel) + big_values = torch.arange(numel, dtype=torch.int32).pin_memory() + + device_copy.begin() + _copy(big[:4], _host([-1, -1, -1, -1])) + _copy(big, big_values) + torch.cuda.synchronize() + + # The staged prefix landed before the full copy, which therefore wins. + assert torch.equal(big.cpu(), big_values) + assert device_copy.is_open() + + +def test_copies_that_together_overflow_the_staging_buffer_all_land(copier): + # Each copy is a quarter of the staging buffer, so the fourth one does not fit. + chunk = _Pack.CAPACITY // 16 + dsts = [_zeros(chunk) for _ in range(6)] + values = [torch.full((chunk,), i + 1, dtype=torch.int32).pin_memory() for i in range(6)] + + device_copy.begin() + for dst, value in zip(dsts, values): + _copy(dst, value) + device_copy.flush() + torch.cuda.synchronize() + + for i, dst in enumerate(dsts): + assert torch.equal(dst.cpu(), values[i]), f"copy {i} did not land" + + +def test_more_copies_than_descriptors_all_land(copier): + count = _Pack.MAX_DESC + 40 + dst = _zeros(count) + + device_copy.begin() + for i in range(count): + _copy(dst[i : i + 1], _host([i + 1])) + device_copy.flush() + torch.cuda.synchronize() + + assert dst.tolist() == list(range(1, count + 1)) + + +def test_a_row_strided_copy_that_needs_more_descriptors_than_remain_falls_back(copier): + rows = 8 + table = torch.full((rows, 4), -1, dtype=torch.int32, device="cuda") + filler = _zeros(_Pack.MAX_DESC - 2) + rows_values = torch.arange(rows * 3, dtype=torch.int32).reshape(rows, 3) + + device_copy.begin() + for i in range(filler.numel()): + _copy(filler[i : i + 1], _host([i + 1])) + # Two descriptors remain; the slice needs one per row, so it flushes and copies directly. + _copy(table[:, :3], rows_values.pin_memory()) + device_copy.flush() + torch.cuda.synchronize() + + assert filler.tolist() == list(range(1, filler.numel() + 1)) + assert torch.equal(table[:, :3].cpu(), rows_values) + assert table[:, 3].tolist() == [-1] * rows + + +def test_mixed_device_copies_past_the_descriptor_limit_all_land(copier): + count = _Pack.MAX_DESC + 10 + src = torch.arange(1, count + 1, dtype=torch.int32, device="cuda") + dst = _zeros(count) + + device_copy.begin() + for i in range(count): + _copy(dst[i : i + 1], src[i : i + 1]) + device_copy.flush() + torch.cuda.synchronize() + + assert torch.equal(dst, src) + + +def test_begin_on_an_open_pack_issues_it_first(copier): + dst = _zeros(2) + + device_copy.begin() + _copy(dst, _host([8, 9])) + device_copy.begin() + torch.cuda.synchronize() + + assert dst.tolist() == [8, 9] + + +def test_reusing_a_pinned_staging_buffer_waits_for_its_upload(copier): + """With every staging buffer in the ring used once, the next pack reuses the first one. + + Its upload is queued behind a long kernel, so rewriting the buffer without waiting would + upload the newer bytes in place of the older ones. + """ + ring = _Pack.RING + dsts = [_zeros(4) for _ in range(ring + 1)] + values = [[i * 10 + j for j in range(4)] for i in range(ring + 1)] + + device_copy.begin() + torch.cuda._sleep(_SLEEP_CYCLES) + _copy(dsts[0], _host(values[0])) + device_copy.flush() + first_upload = copier._pack.events[0] + for i in range(1, ring): + device_copy.begin() + _copy(dsts[i], _host(values[i])) + device_copy.flush() + assert not first_upload.query(), "the queued kernel finished too early for this check" + + device_copy.begin() + assert first_upload.query() + _copy(dsts[ring], _host(values[ring])) + device_copy.flush() + torch.cuda.synchronize() + + for i, dst in enumerate(dsts): + assert dst.tolist() == values[i], f"pack {i} uploaded the wrong bytes" + + +def test_a_copy_while_capturing_is_not_merged(copier): + dst = _zeros(3) + src = torch.tensor([1, 2, 3], dtype=torch.int32, device="cuda") + + device_copy.begin() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + _copy(dst, src) + device_copy.flush() + torch.cuda.synchronize() + assert dst.tolist() == [0, 0, 0] + + graph.replay() + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3] + + +# --------------------------------------------------------------------------- +# Discard +# --------------------------------------------------------------------------- + + +def test_discard_drops_the_staged_copies(copier): + dst = _zeros(3) + + device_copy.begin() + _copy(dst, _host([1, 2, 3])) + _copy(dst, torch.ones(3, dtype=torch.int32, device="cuda")) + device_copy.discard() + pack = copier._pack + assert not pack.open and pack.staged == [] and pack.h2d == [] and pack.d2d == [] + device_copy.flush() + torch.cuda.synchronize() + + assert dst.tolist() == [0, 0, 0] + + +def test_after_a_discarded_step_identical_values_are_sent_again(copier): + """The runner's recovery: discard drops the pack, and on_drop drops the owner's record of the merged copy.""" + dst = _zeros(3) + values = _host([4, 5, 6]) + + device_copy.begin() + assert _skip_copy(dst, values) is True + device_copy.discard() + torch.cuda.synchronize() + assert dst.tolist() == [0, 0, 0] + + device_copy.begin() + assert _skip_copy(dst, values) is True + device_copy.flush() + torch.cuda.synchronize() + + assert dst.tolist() == [4, 5, 6] + + +def test_skipped_copies_join_the_open_pack(copier): + dst = _zeros(3) + values = _host([4, 5, 6]) + + device_copy.begin() + assert _skip_copy(dst, values) is True + torch.cuda.synchronize() + assert dst.tolist() == [0, 0, 0] + # Already staged with these values: the second copy is skipped. + assert _skip_copy(dst, values) is False + device_copy.flush() + torch.cuda.synchronize() + + assert dst.tolist() == [4, 5, 6] + + +# --------------------------------------------------------------------------- +# Skipping unchanged copies +# --------------------------------------------------------------------------- + + +def test_a_repeated_identical_copy_is_skipped(copier): + dst = _zeros(4) + + assert _skip_copy(dst, _host([5, 6, 7, 8])) is True + assert _skip_copy(dst, _host([5, 6, 7, 8])) is False + torch.cuda.synchronize() + assert dst.tolist() == [5, 6, 7, 8] + + +def test_changed_values_are_copied(copier): + dst = _zeros(4) + + assert _skip_copy(dst, _host([1, 2, 3, 4])) is True + assert _skip_copy(dst, _host([1, 2, 3, 5])) is True + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3, 5] + assert _skip_copy(dst, _host([1, 2, 3, 5])) is False + + +def test_the_shadow_is_a_snapshot_of_the_source(copier): + """Mutating the host source after the copy must not make a later copy look unchanged.""" + dst = _zeros(3) + src = _host([1, 2, 3]) + + assert _skip_copy(dst, src) is True + torch.cuda.synchronize() + src[0] = 9 + assert _skip_copy(dst, src) is True + torch.cuda.synchronize() + assert dst.tolist() == [9, 2, 3] + + +def test_a_source_dtype_change_copies(copier): + dst = _zeros(3) + + assert _skip_copy(dst, _host([1, 2, 3], torch.int32)) is True + assert _skip_copy(dst, _host([1, 2, 3], torch.int64)) is True + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3] + + +def test_a_copy_through_another_view_replaces_the_shadow(copier): + buf = _zeros(8) + head, tail = buf[:4], buf[4:] + values = _host([1, 2, 3, 4]) + + assert _skip_copy(head, values) is True + assert _skip_copy(tail, _host([5, 6, 7, 8])) is True + # The shadow now describes the tail, so the head copy must not be skipped. + assert _skip_copy(head, values) is True + assert _skip_copy(head, values) is False + torch.cuda.synchronize() + assert buf.tolist() == [1, 2, 3, 4, 5, 6, 7, 8] + + +def test_an_overlapping_view_never_skips_on_stale_state(copier): + buf = _zeros(6) + values = _host([1, 1, 1, 1]) + + assert _skip_copy(buf[:4], values) is True + assert _skip_copy(buf[2:6], _host([2, 2, 2, 2])) is True + # buf[2:4] was overwritten through the other view: same values, same view, must copy. + assert _skip_copy(buf[:4], values) is True + torch.cuda.synchronize() + assert buf.tolist() == [1, 1, 1, 1, 2, 2] + + +def test_a_view_with_another_shape_of_the_same_bytes_copies(copier): + buf = _zeros(4) + + assert _skip_copy(buf, _host([1, 2, 3, 4])) is True + assert _skip_copy(buf.view(2, 2), _host([[1, 2], [3, 4]])) is True + assert _skip_copy(buf.view(2, 2), _host([[1, 2], [3, 4]])) is False + + +def test_the_owner_forgets_a_record_another_writer_invalidated(copier): + dst = _zeros(4) + values = _host([1, 2, 3, 4]) + + assert _skip_copy(dst, values) is True + # A device write the records do not see: the owner must forget the record, then the copy is sent again. + _copy(dst, torch.arange(4, dtype=torch.int32, device="cuda")) + _REC.forget(dst) + assert _skip_copy(dst, values) is True + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3, 4] + + +def test_a_failed_copy_through_another_view_invalidates_the_old_record(copier, monkeypatch): + dst = _zeros(4) + values = _host([1, 2, 3, 4]) + assert _skip_copy(dst, values) is True + other = _host([9, 9, 9]) + plain_copy = torch.Tensor.copy_ + + def partial_copy_then_fail(target, source, non_blocking=False): + plain_copy(target, source, non_blocking=non_blocking) + raise RuntimeError("copy submission failed") + + with monkeypatch.context() as patch: + patch.setattr(torch.Tensor, "copy_", partial_copy_then_fail) + with pytest.raises(RuntimeError, match="copy submission failed"): + _skip_copy(dst[1:], other) + # The failed copy wrote dst[1:], so the record of the whole view must not vouch for it. + assert _skip_copy(dst, values) is True + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3, 4] + + +def test_capture_always_copies_and_drops_the_record(copier): + dst = _zeros(4) + values = _host([4, 3, 2, 1]) + assert _skip_copy(dst, values) is True + torch.cuda.synchronize() + + graph = torch.cuda.CUDAGraph() + results = [] + with torch.cuda.graph(graph): + results.append(_skip_copy(dst, values)) + assert results == [True] + + dst.zero_() + graph.replay() + torch.cuda.synchronize() + assert dst.tolist() == [4, 3, 2, 1] + # The captured copy reruns on every replay, so the record could not vouch for the buffer. + assert _skip_copy(dst, values) is True + + +def test_unchanged_is_refused_during_capture(copier): + dst = _zeros(4) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + with pytest.raises( + ValueError, match=r"_Site\.SKIP: unchanged=True during CUDA graph capture" + ): + device_copy.copy(_Site.SKIP, dst, _host([1, 1, 1, 1]), unchanged=True) + with pytest.raises( + ValueError, match=r"_Site\.TAIL: unchanged=True during CUDA graph capture" + ): + device_copy.fill_tail(_Site.TAIL, dst, 1, -1, unchanged=True) + dst.add_(1) + + +def test_a_repeated_tail_fill_is_skipped_and_a_moved_start_refills(copier): + buf = _zeros(6) + _REC.fill_tail(_Site.TAIL, buf, 2, -1) + # Writes nothing else is allowed to make; they show which fills ran. + buf[3] = 5 + _REC.fill_tail(_Site.TAIL, buf, 2, -1) + assert buf[3].item() == 5 + _REC.fill_tail(_Site.TAIL, buf, 4, -1) + buf[3] = 7 + _REC.fill_tail(_Site.TAIL, buf, 2, -1) + assert buf.tolist() == [0, 0, -1, -1, -1, -1] + + +def test_a_whole_buffer_write_drops_the_tail_record(copier): + buf = _zeros(6) + _REC.fill_tail(_Site.TAIL, buf, 2, -1) + buf.copy_(torch.arange(6, dtype=torch.int32, device="cuda")) + _REC.fill_tail(_Site.TAIL, buf, 6, -1) + _REC.fill_tail(_Site.TAIL, buf, 2, -1) + assert buf.tolist() == [0, 1, -1, -1, -1, -1] + + +def test_a_tail_fill_is_eager_so_discard_keeps_its_record(copier): + buf = _zeros(6) + device_copy.begin() + _REC.fill_tail(_Site.TAIL, buf, 2, -1) + device_copy.discard() + # The fill ran on the stream, not in the pack: the record still describes the tail. + buf[0] = 3 + _REC.fill_tail(_Site.TAIL, buf, 2, -1) + torch.cuda.synchronize() + assert buf.tolist() == [3, 0, -1, -1, -1, -1] + + +def test_a_tail_record_keeps_its_storage_so_a_new_buffer_cannot_inherit_it(copier): + buf = _zeros(1 << 20) + _REC.fill_tail(_Site.TAIL, buf, 3, -1) + address = buf.data_ptr() + del buf + # Without the record holding the storage, the allocator hands this same block back, and the fill below would be + # skipped on the freed buffer's record. + fresh = _zeros(1 << 20) + assert fresh.data_ptr() != address + _REC.fill_tail(_Site.TAIL, fresh, 3, -1) + assert fresh[3:].eq(-1).all() + + +# --------------------------------------------------------------------------- +# Isolation between copiers +# --------------------------------------------------------------------------- + + +def test_a_nested_copier_does_not_see_the_outer_pack(copier): + outer_dst = _zeros(2) + inner_dst = _zeros(2) + other = _new_copier() + + device_copy.begin() + _copy(outer_dst, _host([1, 2])) + with device_copy.active(other): + assert not device_copy.is_open() + _copy(inner_dst, _host([3, 4])) # plain copy: the inner pack is closed + device_copy.begin() + _copy(inner_dst, _host([5, 6])) + torch.cuda.synchronize() + assert inner_dst.tolist() == [3, 4] + assert outer_dst.tolist() == [0, 0] + device_copy.flush() + assert device_copy.is_open() and len(copier._pack.h2d) == 1 + device_copy.flush() + torch.cuda.synchronize() + + assert inner_dst.tolist() == [5, 6] + assert outer_dst.tolist() == [1, 2] + + +def test_active_none_disables_the_copier_inside_an_outer_one(copier): + dst = _zeros(2) + values = _host([7, 8]) + + device_copy.begin() + with device_copy.active(None): + assert device_copy.current() is None and not device_copy.is_open() + assert _copy(dst, values) is True + torch.cuda.synchronize() + assert dst.tolist() == [7, 8] + assert copier._pack.h2d == [] + + +def test_two_owners_keep_their_own_records(copier): + dst = _zeros(3) + values = _host([7, 8, 9]) + other = _Owner() + + assert _skip_copy(dst, values) is True + # The other owner has never uploaded into dst. + assert other.copy(_Site.SKIP, dst, values) is True + assert other.copy(_Site.SKIP, dst, values) is False + assert _skip_copy(dst, values) is False + + +def test_a_copier_active_on_one_thread_is_invisible_to_another(copier): + mine = _zeros(2) + theirs = _zeros(2) + seen = {} + + device_copy.begin() + _copy(mine, _host([1, 1])) + + def other_thread(): + seen["open"] = device_copy.is_open() + seen["copier"] = device_copy.current() + _copy(theirs, _host([2, 2])) + device_copy.flush() + torch.cuda.synchronize() + seen["theirs"] = theirs.tolist() + seen["mine"] = mine.tolist() + + thread = threading.Thread(target=other_thread) + thread.start() + thread.join() + + assert seen == {"open": False, "copier": None, "theirs": [2, 2], "mine": [0, 0]} + assert device_copy.is_open() and len(copier._pack.h2d) == 1 + device_copy.flush() + torch.cuda.synchronize() + assert mine.tolist() == [1, 1] + + +def test_threads_with_their_own_copiers_keep_them_apart(): + copiers = [_new_copier() for _ in range(2)] + dsts = [_zeros(4) for _ in range(2)] + barrier = threading.Barrier(2) + results = {} + + def worker(index): + with device_copy.active(copiers[index]): + device_copy.begin() + _copy(dsts[index], _host([index + 1] * 4)) + barrier.wait() + # Both packs are open with one staged copy each. + results[index] = (copiers[index].is_open, len(copiers[index]._pack.h2d)) + barrier.wait() + device_copy.flush() + torch.cuda.synchronize() + + threads = [threading.Thread(target=worker, args=(i,)) for i in range(2)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + + assert results == {0: (True, 1), 1: (True, 1)} + assert dsts[0].tolist() == [1] * 4 + assert dsts[1].tolist() == [2] * 4 + + +# --------------------------------------------------------------------------- +# Barriers (flush_after) +# --------------------------------------------------------------------------- + +# _Site.SKIP is merged and a barrier (and its owner skips it when unchanged). +_BARRIER_PLAN = CopyPlan(merge={_Site.H2D, _Site.D2D, _Site.SKIP}, flush_after={_Site.SKIP}) + + +@pytest.fixture +def barrier_copier(): + active = _new_copier(_BARRIER_PLAN) + with device_copy.active(active): + yield active + device_copy.flush() + torch.cuda.synchronize() + + +def test_flush_after_is_validated(): + with pytest.raises(ValueError, match="must be in merge"): + CopyPlan(flush_after={_Site.SKIP}) + with pytest.raises(ValueError, match="must be in merge"): + CopyPlan(merge={_Site.H2D}, flush_after={_Site.PLAIN}) + with pytest.raises(TypeError, match="CopySiteEnum"): + CopyPlan(merge={_Site.H2D}, flush_after={"test.skip"}) + plan = CopyPlan(merge={_Site.H2D, _Site.SKIP}, flush_after={_Site.SKIP}) + assert plan.flush_after == frozenset({_Site.SKIP}) and plan + + +def test_a_staged_anchor_lands_with_its_segment(barrier_copier): + before, anchor, after = _zeros(2), _zeros(2), _zeros(2) + pack = barrier_copier._pack + + device_copy.begin() + slot = pack.slot + _copy(before, _host([1, 2])) + assert _skip_copy(anchor, _host([3, 4])) is True + torch.cuda.synchronize() + # The anchor's own copy has landed (guaranteed); so has what was staged in its segment (as implemented, not + # promised: a reader of ``before`` would anchor on its own site). The pack stays open. + assert (before.tolist(), anchor.tolist()) == ([1, 2], [3, 4]) + assert device_copy.is_open() and pack.seg > 0 + _copy(after, _host([5, 6])) + torch.cuda.synchronize() + assert after.tolist() == [0, 0] + device_copy.flush() + torch.cuda.synchronize() + assert after.tolist() == [5, 6] + assert pack.slot == slot + + +def test_a_skipped_anchor_does_not_flush(barrier_copier): + anchor, merged = _zeros(2), _zeros(2) + pack = barrier_copier._pack + + device_copy.begin() + _skip_copy(anchor, _host([3, 4])) + device_copy.flush() + + device_copy.begin() + _copy(merged, _host([7, 8])) + assert _skip_copy(anchor, _host([3, 4])) is False + torch.cuda.synchronize() + # The anchor had nothing of its own to land: the pack was not flushed. + assert merged.tolist() == [0, 0] and pack.seg == 0 + device_copy.flush() + torch.cuda.synchronize() + assert merged.tolist() == [7, 8] + + +def test_an_eager_or_unstageable_anchor_does_not_add_a_barrier(barrier_copier): + merged, anchor = _zeros(2), _zeros(8) + pack = barrier_copier._pack + + # Pack closed: the anchor runs eagerly, nothing is staged, nothing flushes. + assert _skip_copy(anchor[:2], _host([1, 1])) is True + assert pack.seg == 0 and not device_copy.is_open() + + device_copy.begin() + _copy(merged, _host([2, 2])) + # A strided anchor cannot be staged: what is staged lands first (order), then the anchor eagerly. + assert device_copy.copy(_Site.SKIP, anchor[::2], _host([5, 5, 5, 5])) is True + torch.cuda.synchronize() + assert merged.tolist() == [2, 2] and anchor[::2].tolist() == [5, 5, 5, 5] + # No barrier segment was added after the eager anchor: the next merged copy waits for the flush. + _copy(merged, _host([3, 3])) + torch.cuda.synchronize() + assert merged.tolist() == [2, 2] + device_copy.flush() + torch.cuda.synchronize() + assert merged.tolist() == [3, 3] + + +def test_a_repeated_anchor_flushes_on_every_call(barrier_copier): + dsts = [_zeros(1) for _ in range(3)] + anchor = _zeros(1) + pack = barrier_copier._pack + + device_copy.begin() + slot, segments = pack.slot, [] + for i, dst in enumerate(dsts): + _copy(dst, _host([i + 1])) + _skip_copy(anchor, _host([10 + i])) + segments.append(pack.seg) + torch.cuda.synchronize() + assert [d.item() for d in dsts[: i + 1]] == list(range(1, i + 2)) + assert anchor.item() == 10 + i + device_copy.flush() + # Three segments, appended one after another in the same ring slot. + assert segments == sorted(set(segments)) and pack.slot == slot + + +def test_segments_of_one_step_do_not_wait_on_the_device(barrier_copier): + """Barriers append segments in the step's slot: no host wait on the queued uploads.""" + dsts = [_zeros(2) for _ in range(3)] + anchor = _zeros(1) + done = torch.cuda.Event() + + device_copy.begin() + torch.cuda._sleep(_SLEEP_CYCLES) + done.record() + for i, dst in enumerate(dsts): + _copy(dst, _host([i, i])) + _skip_copy(anchor, _host([i])) + device_copy.flush() + assert not done.query(), ( + "a barrier waited for the device (or the queued kernel finished too early)" + ) + torch.cuda.synchronize() + assert [d.tolist() for d in dsts] == [[0, 0], [1, 1], [2, 2]] + + +def test_capacity_runs_out_across_segments_and_falls_back_in_order(barrier_copier): + # Each chunk is a quarter of the staging buffer: two segments hold one each, the third does not fit. + chunk = _Pack.CAPACITY // 16 + dsts = [_zeros(chunk) for _ in range(3)] + values = [torch.full((chunk,), i + 1, dtype=torch.int32).pin_memory() for i in range(3)] + anchor = _zeros(1) + + device_copy.begin() + for i in range(2): + _copy(dsts[i], values[i]) + _skip_copy(anchor, _host([i])) + # A small merged write into the last chunk, then the whole chunk, which no longer fits: the small write is issued + # first and the eager copy of the chunk wins. + _copy(dsts[2][:4], _host([-1, -1, -1, -1])) + _copy(dsts[2], values[2]) + assert device_copy.is_open() + device_copy.flush() + torch.cuda.synchronize() + for i, dst in enumerate(dsts): + assert torch.equal(dst.cpu(), values[i]), f"chunk {i}" + + +def test_host_and_device_copies_keep_their_order_across_segments(barrier_copier): + dst = _zeros(2) + mirror = _zeros(2) + src = torch.tensor([7, 7], dtype=torch.int32, device="cuda") + anchor = _zeros(1) + + device_copy.begin() + # Segment 1: a device copy into dst. Segment 2: a host copy into dst, then a device copy reading it. + _copy(dst, src) + _skip_copy(anchor, _host([1])) + _copy(dst, _host([3, 4])) + _copy(mirror, dst) + device_copy.flush() + torch.cuda.synchronize() + assert dst.tolist() == [3, 4] and mirror.tolist() == [3, 4] + + +# --------------------------------------------------------------------------- +# Check mode constant and expect_landed +# --------------------------------------------------------------------------- + + +def test_a_copier_takes_check_mode_from_the_module_constant(monkeypatch): + monkeypatch.setattr(device_copy, "CHECK_MODE", True) + assert _new_copier().check and _new_copier()._pack.check + monkeypatch.setattr(device_copy, "CHECK_MODE", False) + assert not _new_copier().check + + +def test_expect_landed_raises_outside_check_mode(monkeypatch): + monkeypatch.setattr(device_copy, "CHECK_MODE", False) + with pytest.raises(RuntimeError, match=r"_Site\.SKIP.*CHECK_MODE"): + device_copy.expect_landed(_Site.SKIP, _zeros(1)) + + +def test_expect_landed_passes_behind_its_barrier(monkeypatch): + monkeypatch.setattr(device_copy, "CHECK_MODE", True) + table = _zeros(4) + with device_copy.active(_new_copier(_BARRIER_PLAN)): + device_copy.begin() + _skip_copy(table, _host([1, 2, 3, 4])) + if device_copy.CHECK_MODE: + device_copy.expect_landed(_Site.SKIP, table) + assert table.sum().item() == 10 + device_copy.flush() + # Without an active copier nothing is staged. + device_copy.expect_landed(_Site.SKIP, table) + + +def test_expect_landed_names_its_anchor_when_the_barrier_is_missing(monkeypatch): + monkeypatch.setattr(device_copy, "CHECK_MODE", True) + table = _zeros(4) + with device_copy.active(_new_copier(CopyPlan(merge={_Site.H2D, _Site.SKIP}))): + device_copy.begin() + _copy(table, _host([1, 2, 3, 4])) + with pytest.raises(RuntimeError, match=rf"{CHECK_ENV}: .*_Site\.SKIP.*_Site\.H2D"): + device_copy.expect_landed(_Site.SKIP, table[1:3]) + # Unrelated bytes are fine. + device_copy.expect_landed(_Site.SKIP, _zeros(4)) + device_copy.discard() + + +# --------------------------------------------------------------------------- +# Owner-side skipping: unchanged and on_drop +# --------------------------------------------------------------------------- + + +def test_on_drop_runs_for_the_staged_copies_of_a_discarded_pack(copier): + dropped = [] + staged, eager = _zeros(2), _zeros(2) + + device_copy.begin() + # A copy that runs eagerly has landed: its on_drop is never kept. + assert device_copy.copy( + _Site.PLAIN, eager, _host([2, 2]), on_drop=lambda: dropped.append("eager") + ) + assert device_copy.copy( + _Site.H2D, staged, _host([1, 1]), on_drop=lambda: dropped.append("staged") + ) + assert not device_copy.copy( + _Site.H2D, staged, _host([3, 3]), unchanged=True, on_drop=lambda: dropped.append("x") + ) + assert dropped == [] + device_copy.discard() + assert dropped == ["staged"] + # A flushed pack never calls them. + device_copy.begin() + device_copy.copy(_Site.H2D, staged, _host([4, 4]), on_drop=lambda: dropped.append("flushed")) + device_copy.flush() + device_copy.discard() + torch.cuda.synchronize() + assert dropped == ["staged"] and staged.tolist() == [4, 4] and eager.tolist() == [2, 2] + + +def test_check_mode_verifies_unchanged(monkeypatch): + monkeypatch.setattr(device_copy, "CHECK_MODE", True) + dst, tail = _zeros(3), _zeros(4) + with device_copy.active(_new_copier()): + device_copy.copy(_Site.PLAIN, dst, _host([1, 2, 3])) + tail[2:].fill_(-1) + # Holding the values: passes. + assert device_copy.copy(_Site.PLAIN, dst, _host([1, 2, 3]), unchanged=True) is False + device_copy.fill_tail(_Site.TAIL, tail, 2, -1, unchanged=True) + with pytest.raises( + RuntimeError, match=rf"{CHECK_ENV}: copy _Site\.PLAIN was passed unchanged=True" + ): + device_copy.copy(_Site.PLAIN, dst, _host([1, 2, 4]), unchanged=True) + with pytest.raises( + RuntimeError, match=rf"{CHECK_ENV}: tail fill _Site\.TAIL was passed unchanged=True" + ): + device_copy.fill_tail(_Site.TAIL, tail, 1, -1, unchanged=True) + # A staged copy still writes the bytes: the device cannot be compared yet, so the check is skipped. + device_copy.begin() + device_copy.copy(_Site.H2D, dst, _host([7, 7, 7])) + assert device_copy.copy(_Site.SKIP, dst, _host([7, 7, 7]), unchanged=True) is False + device_copy.flush() + torch.cuda.synchronize() + assert dst.tolist() == [7, 7, 7] diff --git a/tests/unittest/_torch/pyexecutor/test_device_copy_changed_span.py b/tests/unittest/_torch/pyexecutor/test_device_copy_changed_span.py new file mode 100644 index 000000000000..9cd931a760c1 --- /dev/null +++ b/tests/unittest/_torch/pyexecutor/test_device_copy_changed_span.py @@ -0,0 +1,240 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""DeviceCopyBuffer.update(..., changed_span=True) through a site in the plan's changed_span: a changed merged 2-D +upload stages only each row's changed span.""" + +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor import device_copy +from tensorrt_llm._torch.pyexecutor.device_copy import ( + CopyPlan, + CopySiteEnum, + DeviceCopier, + DeviceCopyBuffer, + Direction, +) + +pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required") + +ROWS, WIDTH = 4, 1024 + + +class _Site(CopySiteEnum): + TABLE = ("test_changed_span.table", Direction.H2D) + + +@pytest.fixture(autouse=True) +def _no_check_mode(monkeypatch): + monkeypatch.setattr(device_copy, "CHECK_MODE", False) + + +def _copier(changed_span: bool = True) -> DeviceCopier: + plan = CopyPlan( + merge={_Site.TABLE}, + changed_span={_Site.TABLE} if changed_span else set(), + parallel_scatter=True, + ) + return DeviceCopier(plan, torch.device("cuda", torch.cuda.current_device())) + + +@pytest.fixture +def pack(): + return _copier() + + +def _record(buf: torch.Tensor) -> DeviceCopyBuffer: + return DeviceCopyBuffer(buf, site=_Site.TABLE) + + +def _table(seed, rows=ROWS, width=WIDTH): + generator = torch.Generator().manual_seed(seed) + return torch.randint( + 0, 1 << 20, (rows, width), generator=generator, dtype=torch.int32 + ).pin_memory() + + +def _upload(copier, record, dst, src): + """One step: open the pack, upload through the record, issue the pack; returns the staged copies.""" + with device_copy.active(copier): + device_copy.begin() + copied = record.update(src, view=dst, changed_span=True) + staged = list(copier._pack.h2d) + device_copy.flush() + torch.cuda.synchronize() + return copied, staged + + +@pytest.mark.parametrize("changed_span", [False, True]) +def test_device_rows_equal_the_source_after_every_step(changed_span): + """Bitwise: whatever is staged, the device view holds the source after each step.""" + pack = _copier(changed_span) + buf = torch.full((8, 2 * WIDTH), -7, dtype=torch.int32, device="cuda") + cache = _record(buf) + dst = buf[:ROWS, :WIDTH] + src = _table(0) + generator = torch.Generator().manual_seed(1) + for step in range(40): + src = src.clone().pin_memory() + # A request crosses into a new page: one entry of one row changes (sometimes several, in several rows). + for _ in range(1 + step % 3): + row = int(torch.randint(0, ROWS, (1,), generator=generator)) + col = int(torch.randint(0, WIDTH, (1,), generator=generator)) + src[row, col] = step + _upload(pack, cache, dst, src) + assert torch.equal(dst.cpu(), src) + # Rows and columns outside the view were never written. + assert bool((buf[ROWS:] == -7).all()) and bool((buf[:, WIDTH:] == -7).all()) + + +def test_one_changed_entry_stages_one_element(pack): + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + cache = _record(dst) + src = _table(2) + _upload(pack, cache, dst, src) + + src = src.clone().pin_memory() + src[2, 517] += 1 + copied, staged = _upload(pack, cache, dst, src) + + assert copied is True + assert staged == [(dst[2, 517].data_ptr(), staged[0][1], 4)] + assert torch.equal(dst.cpu(), src) + + +def test_each_changed_row_stages_its_first_to_last_changed_column(pack): + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + cache = _record(dst) + src = _table(3) + _upload(pack, cache, dst, src) + + src = src.clone().pin_memory() + src[0, 10] += 1 + src[0, 20] += 1 + src[3, 1000] += 1 + _, staged = _upload(pack, cache, dst, src) + + assert [(ptr, nbytes) for ptr, _, nbytes in staged] == [ + (dst[0, 10].data_ptr(), 11 * 4), + (dst[3, 1000].data_ptr(), 4), + ] + assert torch.equal(dst.cpu(), src) + + +def test_wider_view_uploads_the_columns_the_shadow_does_not_cover(pack): + buf = torch.full((ROWS, WIDTH), -1, dtype=torch.int32, device="cuda") + cache = _record(buf) + src = _table(4) + _upload(pack, cache, buf[:, : WIDTH // 2], src[:, : WIDTH // 2].contiguous().pin_memory()) + + _, staged = _upload( + pack, cache, buf[:, : WIDTH // 2 + 3], src[:, : WIDTH // 2 + 3].contiguous().pin_memory() + ) + + assert [nbytes for _, _, nbytes in staged] == [3 * 4] * ROWS + assert torch.equal(buf[:, : WIDTH // 2 + 3].cpu(), src[:, : WIDTH // 2 + 3]) + + +def test_a_mostly_changed_table_is_staged_whole(pack): + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + cache = _record(dst) + _upload(pack, cache, dst, _table(5)) + + src = _table(6) + _, staged = _upload(pack, cache, dst, src) + + assert [nbytes for _, _, nbytes in staged] == [ROWS * WIDTH * 4] + assert torch.equal(dst.cpu(), src) + + +def test_a_site_outside_the_plans_changed_span_stages_the_whole_view(): + pack = _copier(changed_span=False) + assert not CopyPlan().changed_span + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + cache = _record(dst) + src = _table(7) + _upload(pack, cache, dst, src) + + src = src.clone().pin_memory() + src[1, 1] += 1 + _, staged = _upload(pack, cache, dst, src) + + assert [nbytes for _, _, nbytes in staged] == [ROWS * WIDTH * 4] + + +def test_an_unmerged_copy_copies_the_whole_view(pack, monkeypatch): + """No open pack: one eager copy of the view, not one per row span.""" + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + cache = _record(dst) + src = _table(8) + with device_copy.active(pack): + cache.update(src, view=dst, changed_span=True) + src = src.clone().pin_memory() + src[1, 1] += 1 + copies = [] + real = device_copy.copy + monkeypatch.setattr( + device_copy, "copy", lambda *a, **k: copies.append(tuple(a[1].shape)) or real(*a, **k) + ) + assert cache.update(src, view=dst, changed_span=True) is True + torch.cuda.synchronize() + assert copies == [(ROWS, WIDTH)] + assert torch.equal(dst.cpu(), src) + + +def test_a_view_with_another_row_stride_is_staged_whole(pack): + """The shadow describes other device bytes: nothing of it may be trusted.""" + buf = torch.zeros((2 * ROWS, WIDTH), dtype=torch.int32, device="cuda") + cache = _record(buf) + src = _table(9) + _upload(pack, cache, buf[:ROWS], src) + + _, staged = _upload(pack, cache, buf[::2], src) + + assert [nbytes for _, _, nbytes in staged] == [WIDTH * 4] * ROWS + assert torch.equal(buf[::2].cpu(), src) + + +def test_a_discarded_span_upload_drops_the_record(pack): + """A merged span copy whose pack is discarded never lands: the record is dropped, so the next step sends all.""" + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + cache = _record(dst) + src = _table(10) + _upload(pack, cache, dst, src) + changed = src.clone().pin_memory() + changed[0, 3] += 1 + with device_copy.active(pack): + device_copy.begin() + cache.update(changed, view=dst, changed_span=True) + device_copy.discard() + assert cache.host_values is None + _, staged = _upload(pack, cache, dst, changed) + assert [nbytes for _, _, nbytes in staged] == [ROWS * WIDTH * 4] + assert torch.equal(dst.cpu(), changed) + + +def test_a_barrier_site_stages_the_whole_view_with_one_barrier(): + """Per-row spans through a flush_after site would each flush on their own: the whole view goes instead.""" + plan = CopyPlan( + merge={_Site.TABLE}, + flush_after={_Site.TABLE}, + changed_span={_Site.TABLE}, + parallel_scatter=True, + ) + copier = DeviceCopier(plan, torch.device("cuda", torch.cuda.current_device())) + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + cache = _record(dst) + src = _table(11) + with device_copy.active(copier): + device_copy.begin() + cache.update(src, view=dst, changed_span=True) + changed = src.clone().pin_memory() + changed[1, 7] += 1 + segments_before = copier._pack.seg + cache.update(changed, view=dst, changed_span=True) + # One more segment (the barrier after the whole view), not one per span. + assert copier._pack.seg > segments_before + assert not copier._pack.h2d + device_copy.flush() + torch.cuda.synchronize() + assert torch.equal(dst.cpu(), changed) diff --git a/tests/unittest/_torch/pyexecutor/test_device_copy_coexistence.py b/tests/unittest/_torch/pyexecutor/test_device_copy_coexistence.py new file mode 100644 index 000000000000..9600f2c10929 --- /dev/null +++ b/tests/unittest/_torch/pyexecutor/test_device_copy_coexistence.py @@ -0,0 +1,114 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The copy plan is opt-in: without a copier the shared call sites take their step-input pack path unchanged, and a +model uses one path or the other, never both.""" + +from types import SimpleNamespace + +import numpy as np +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor import device_copy, step_inputs +from tensorrt_llm._torch.pyexecutor.cuda_graph_runner import CUDAGraphRunner +from tensorrt_llm._torch.pyexecutor.device_copy import CopyPlan, CopySite, DeviceCopier +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder.runner import check_one_copy_path +from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks +from tensorrt_llm._torch.pyexecutor.step_inputs import StepInputPack +from tensorrt_llm._torch.speculative.interface import _upload_host_values + +_PLAN = CopyPlan(merge={CopySite.GRAPH_INPUT_IDS, CopySite.SPEC_RNG_SEEDS}) + + +def _scatter_op_registered() -> bool: + return torch.cuda.is_available() and hasattr(torch.ops.trtllm, "step_inputs_scatter") + + +_needs_scatter = pytest.mark.skipif( + not _scatter_op_registered(), + reason="CUDA and torch.ops.trtllm.step_inputs_scatter (built from " + "cpp/tensorrt_llm/thop/minimaxM3/stepInputsScatterOp.cpp) are required", +) + + +def _device(): + return torch.device("cuda", torch.cuda.current_device()) + + +@pytest.mark.parametrize( + "hooks", [RuntimeHooks(pack_step_inputs=True), RuntimeHooks(skip_unchanged_h2d=True)] +) +def test_a_model_with_a_plan_cannot_also_turn_on_the_step_input_pack(hooks): + with pytest.raises(ValueError, match="as a whole"): + check_one_copy_path(SimpleNamespace(), _PLAN, hooks) + + +@pytest.mark.parametrize( + "plan, hooks", + [ + (CopyPlan(), RuntimeHooks(pack_step_inputs=True, skip_unchanged_h2d=True)), + (_PLAN, RuntimeHooks()), + (CopyPlan(), RuntimeHooks()), + ], +) +def test_either_path_alone_is_accepted(plan, hooks): + check_one_copy_path(SimpleNamespace(), plan, hooks) + + +@pytest.mark.parametrize("pack", [False, True]) +def test_a_routed_upload_without_a_copier_keeps_its_pack_choice(monkeypatch, pack): + calls = [] + monkeypatch.setattr(step_inputs, "copy_", lambda dst, src: calls.append("pack")) + monkeypatch.setattr(device_copy, "copy", lambda *args, **kwargs: calls.append("plan")) + dst = torch.zeros(4, dtype=torch.int64) + _upload_host_values(dst, np.arange(3, dtype=np.int64), pack, CopySite.SPEC_RNG_SEEDS) + if pack: + assert calls == ["pack"] + else: + assert calls == [] and dst.tolist() == [0, 1, 2, 0] + + +@pytest.mark.parametrize("pack", [False, True]) +def test_a_routed_upload_with_a_copier_goes_through_its_site(monkeypatch, pack): + calls = [] + monkeypatch.setattr(step_inputs, "copy_", lambda dst, src: calls.append("pack")) + monkeypatch.setattr(device_copy, "copy", lambda site, dst, src: calls.append(site)) + monkeypatch.setattr(device_copy, "current", lambda: object()) + _upload_host_values( + torch.zeros(4, dtype=torch.int64), + np.arange(3, dtype=np.int64), + pack, + CopySite.SPEC_RNG_SEEDS, + ) + assert calls == [CopySite.SPEC_RNG_SEEDS] + + +@_needs_scatter +def test_a_static_input_copy_without_a_copier_is_staged_into_the_open_step_input_pack(): + dst = torch.zeros(4, dtype=torch.int32, device="cuda") + src = torch.arange(4, dtype=torch.int32, device="cuda") + pack = StepInputPack(_device()) + with step_inputs.active(pack): + step_inputs.begin() + CUDAGraphRunner._copy_static_input(CopySite.GRAPH_INPUT_IDS, dst, src) + torch.cuda.synchronize() + # Staged into the pack, not issued: it lands at the pack's flush. + assert dst.tolist() == [0, 0, 0, 0] + step_inputs.flush() + torch.cuda.synchronize() + assert dst.tolist() == [0, 1, 2, 3] + + +@_needs_scatter +def test_a_static_input_copy_with_a_copier_is_merged_through_its_site(): + dst = torch.zeros(4, dtype=torch.int32, device="cuda") + src = torch.arange(4, dtype=torch.int32, device="cuda") + copier = DeviceCopier(_PLAN, _device()) + with device_copy.active(copier): + device_copy.begin() + CUDAGraphRunner._copy_static_input(CopySite.GRAPH_INPUT_IDS, dst, src) + assert copier.pending_sites(dst) == [CopySite.GRAPH_INPUT_IDS] + assert not step_inputs.is_open() + device_copy.flush() + torch.cuda.synchronize() + assert dst.tolist() == [0, 1, 2, 3] diff --git a/tests/unittest/_torch/pyexecutor/test_device_copy_in_graph.py b/tests/unittest/_torch/pyexecutor/test_device_copy_in_graph.py new file mode 100644 index 000000000000..11d37696d5eb --- /dev/null +++ b/tests/unittest/_torch/pyexecutor/test_device_copy_in_graph.py @@ -0,0 +1,535 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""A copy plan's parallel scatter and its in-graph final flush (``CopyPlan.parallel_scatter``, ``InGraph``).""" + +import time +import weakref +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor import device_copy +from tensorrt_llm._torch.pyexecutor.cuda_graph_runner import CUDAGraphRunner, KeyType +from tensorrt_llm._torch.pyexecutor.device_copy import ( + CopyPlan, + CopySiteEnum, + DeviceCopier, + Direction, + InGraph, +) + + +def _scatter_op_registered() -> bool: + return torch.cuda.is_available() and hasattr(torch.ops.trtllm, "step_inputs_scatter") + + +pytestmark = [ + pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required"), + pytest.mark.skipif( + not _scatter_op_registered(), + reason="torch.ops.trtllm.step_inputs_scatter is not registered (built from " + "cpp/tensorrt_llm/thop/minimaxM3/stepInputsScatterOp.cpp)", + ), +] + +# About half a second of GPU time: long enough that work queued behind it is still pending when +# the host checks, short enough for a unit test. +_SLEEP_CYCLES = 1_000_000_000 + + +class _Site(CopySiteEnum): + H2D = ("test_in_graph.h2d", Direction.H2D) + D2D = ("test_in_graph.d2d", Direction.D2D) + + +@pytest.fixture(autouse=True) +def _no_check_mode(monkeypatch): + # These tests read merged destinations directly; check mode (on for unit tests) would poison them. + monkeypatch.setattr(device_copy, "CHECK_MODE", False) + + +def _device(): + return torch.device("cuda", torch.cuda.current_device()) + + +def _host(values, dtype=torch.int32): + return torch.tensor(values, dtype=dtype).pin_memory() + + +def _zeros(*shape, dtype=torch.int32): + return torch.zeros(*shape, dtype=dtype, device="cuda") + + +def _copier(parallel_scatter=False, final_flush=None) -> DeviceCopier: + plan = CopyPlan( + merge={_Site.H2D, _Site.D2D}, parallel_scatter=parallel_scatter, final_flush=final_flush + ) + return DeviceCopier(plan, _device()) + + +def _copy(dst, src): + device_copy.copy(_Site.D2D if src.is_cuda else _Site.H2D, dst, src) + + +# --------------------------------------------------------------------------- +# Parallel scatter +# --------------------------------------------------------------------------- + + +def _random_pack_run(parallel: bool, seed: int, overlap: bool): + """Stage a fixed random mix of host copies (2-D page-table rows included) and device copies that read a host + copy's destination; return every destination after the flush.""" + g = torch.Generator().manual_seed(seed) + copier = _copier(parallel_scatter=parallel) + big = _zeros(4096) + table = torch.zeros((8, 64), dtype=torch.int32, device=_device()) + out = _zeros(256) + with device_copy.active(copier): + device_copy.begin() + for k in range(24): + n = int(torch.randint(1, 64, (1,), generator=g)) + _copy( + big[k * 128 : k * 128 + n], + torch.randint(0, 1 << 30, (n,), generator=g, dtype=torch.int32), + ) + _copy(table[:3, :40], torch.randint(0, 1 << 30, (3, 40), generator=g, dtype=torch.int32)) + if overlap: + # A later host copy overwrites part of an earlier one: order matters, so the serial scatter must run. + _copy(big[0:4], torch.full((4,), -7, dtype=torch.int32)) + _copy(out[:64], big[:64]) + _copy(out[64:96], big[128:160]) + device_copy.flush() + torch.cuda.synchronize() + return big.cpu(), table.cpu(), out.cpu() + + +@pytest.mark.parametrize("overlap", [False, True], ids=["disjoint", "overlapping"]) +@pytest.mark.parametrize("seed", [0, 1, 2]) +def test_parallel_scatter_writes_what_the_serial_scatter_writes(seed, overlap): + serial = _random_pack_run(False, seed, overlap) + parallel = _random_pack_run(True, seed, overlap) + for a, b in zip(serial, parallel): + assert torch.equal(a, b) + + +def _compiled_variants(kernel) -> int: + """Compiled variants of a Triton kernel on this device, or -1 where this Triton keeps them elsewhere.""" + caches = getattr(kernel, "device_caches", None) + if not caches: + return -1 + cache = caches.get(torch.cuda.current_device()) + return len(cache[0]) if cache else 0 + + +@pytest.mark.parametrize("count", [1, 15, 16, 17, 120]) # 120: 240 of the 256 descriptors +def test_parallel_scatter_is_bitwise_for_every_copy_count(count): + """``first`` (the device copies' descriptor offset) is the segment's host-copy count: one compiled variant serves + every count, so a new count never compiles mid-serving.""" + copier = _copier(parallel_scatter=True) + pack = copier._pack + width = 3 + dst = _zeros(count * width) + mirror = _zeros(count * width) + g = torch.Generator().manual_seed(count) + values = torch.randint( + -(1 << 31), (1 << 31) - 1, (count, width), generator=g, dtype=torch.int32 + ) + with device_copy.active(copier): + device_copy.begin() + for i in range(count): + _copy(dst[i * width : (i + 1) * width], values[i]) + for i in range(count): + _copy(mirror[i * width : (i + 1) * width], dst[i * width : (i + 1) * width]) + assert len(pack.h2d) == count and len(pack.d2d) == count + device_copy.flush() + torch.cuda.synchronize() + assert torch.equal(dst.cpu(), values.flatten()) + assert torch.equal(mirror.cpu(), values.flatten()) + variants = _compiled_variants(device_copy._parallel_scatter_kernel) + assert variants in (-1, 1, 2), variants # at most one per DEVICE_SRC + + +def test_parallel_scatter_only_for_order_free_packs(): + assert device_copy._parallel_ok([(0, 0, 8), (8, 0, 8)], []) + assert not device_copy._parallel_ok([(0, 0, 8), (4, 0, 8)], []) + # A device copy may read a host copy's destination (the launches run in order) ... + assert device_copy._parallel_ok([(0, 0, 8)], [(100, 0, 8)]) + # ... but not another device copy's destination, and device destinations must be disjoint. + assert not device_copy._parallel_ok([], [(100, 0, 8), (200, 100, 8)]) + assert not device_copy._parallel_ok([], [(100, 0, 8), (104, 300, 8)]) + + +def test_parallel_scatter_is_off_by_default(): + assert not CopyPlan().parallel_scatter + assert not _copier()._pack.parallel_scatter + + +def test_a_barrier_segment_scatters_in_parallel_too(): + """Every segment of a step takes the parallel scatter when order-free, not only the last one.""" + plan = CopyPlan(merge={_Site.H2D}, flush_after={_Site.H2D}, parallel_scatter=True) + copier = DeviceCopier(plan, _device()) + a, b = _zeros(4), _zeros(4) + with device_copy.active(copier): + device_copy.begin() + _copy(a, _host([1, 2, 3, 4])) + _copy(b, _host([5, 6, 7, 8])) + device_copy.flush() + torch.cuda.synchronize() + assert a.tolist() == [1, 2, 3, 4] and b.tolist() == [5, 6, 7, 8] + + +# --------------------------------------------------------------------------- +# In-graph final flush +# --------------------------------------------------------------------------- + + +def _graph_case(early=False, epoch=False): + """A copier with the in-graph prologue and a graph that starts with it and then reads the destinations.""" + copier = _copier(parallel_scatter=True, final_flush=InGraph(early_copy=early, step_epoch=epoch)) + a, b, c = _zeros(16), _zeros(16), _zeros(16) + out = _zeros(48) + graph = torch.cuda.CUDAGraph() + with device_copy.active(copier): + torch.cuda.synchronize() + with torch.cuda.graph(graph): + assert device_copy.capture_prologue() is copier + out.copy_(torch.cat([a, b, c])) + return copier, graph, (a, b, c), out + + +def _step(copier, graph, dsts, values, overlap=False, flush=True): + a, b, c = dsts + with device_copy.active(copier): + device_copy.begin() + _copy(a, _host(values[0])) + _copy(b, _host(values[1])) + if overlap: + _copy(a[:4], _host([-1, -2, -3, -4])) + _copy(c, a) # a device copy reading a host copy's destination + if not flush: + device_copy.discard() + device_copy.flush(in_graph=True) + graph.replay() + copier.on_graph_replay() + + +def test_in_graph_pack_is_issued_by_the_replay(): + copier, graph, dsts, out = _graph_case() + for step in range(5): + va = [step * 100 + i for i in range(16)] + vb = [step * 1000 - i for i in range(16)] + _step(copier, graph, dsts, (va, vb)) + torch.cuda.synchronize() + assert out.tolist() == va + vb + va, step + # Nothing was issued outside the graph: the window carried the copies. + assert int(copier._pack.ghost[:8].view(torch.int64)[0]) == 2 + + +def test_in_graph_pack_not_order_free_is_issued_eagerly(): + copier, graph, dsts, out = _graph_case() + va = list(range(16)) + vb = list(range(100, 116)) + _step(copier, graph, dsts, (va, vb), overlap=True) + torch.cuda.synchronize() + expect_a = [-1, -2, -3, -4] + va[4:] + assert out.tolist() == expect_a + vb + expect_a + # The window was zeroed, so the prologue wrote nothing over the eager result. + assert not copier._pack.ghost[:16].any() + + +def test_in_graph_replay_without_a_pack_reapplies_nothing(): + copier, graph, dsts, out = _graph_case() + _step(copier, graph, dsts, (list(range(16)), list(range(16)))) + torch.cuda.synchronize() + dsts[0].fill_(7) + _step(copier, graph, dsts, ([0] * 16, [0] * 16), flush=False) + torch.cuda.synchronize() + assert dsts[0].tolist() == [7] * 16 + + +@pytest.mark.parametrize("early", [False, True], ids=["prologue_copy", "early_copy"]) +def test_in_graph_pack_lands_every_step(early): + copier, graph, dsts, out = _graph_case(early) + assert copier._pack.early_copy is early + for step in range(6): + va = [step * 100 + i for i in range(16)] + vb = [step * 1000 - i for i in range(16)] + _step(copier, graph, dsts, (va, vb), overlap=step % 2 == 1) + torch.cuda.synchronize() + head = [-1, -2, -3, -4] if step % 2 == 1 else [] + expect_a = head + va[len(head) :] + assert out.tolist() == expect_a + vb + expect_a, step + + +def test_only_the_last_segment_rides_the_window(): + """A barrier segment is issued eagerly before the replay; the window carries what is staged after it.""" + plan = CopyPlan( + merge={_Site.H2D, _Site.D2D}, + flush_after={_Site.D2D}, + parallel_scatter=True, + final_flush=InGraph(early_copy=True), + ) + copier = DeviceCopier(plan, _device()) + a, b, c = _zeros(16), _zeros(16), _zeros(16) + out = _zeros(48) + graph = torch.cuda.CUDAGraph() + with device_copy.active(copier): + torch.cuda.synchronize() + with torch.cuda.graph(graph): + device_copy.capture_prologue() + out.copy_(torch.cat([a, b, c])) + for step in range(3): + va = [step * 10 + i for i in range(16)] + vb = [step * 100 + i for i in range(16)] + with device_copy.active(copier): + device_copy.begin() + _copy(a, _host(va)) + _copy(c, a) # a barrier: a and c land now, eagerly + barrier_seen = c.clone() + _copy(b, _host(vb)) + device_copy.flush(in_graph=True) + # One host copy in the window. + assert int(copier._pack.ghost[:8].view(torch.int64)[0]) == 1 + graph.replay() + copier.on_graph_replay() + torch.cuda.synchronize() + assert barrier_seen.tolist() == va, step + assert out.tolist() == va + vb + va, step + + +def _history_run(copier, graph, dsts, out, steps, before_replay=None, before_upload=None): + """Steps back to back without host syncs; each replay's output is copied to a history row behind it (same + stream), so a later upload that overwrote the window too early shows up as the wrong step's values.""" + hist = _zeros(steps, out.numel()) + expect = [] + for step in range(steps): + va = [step * 100 + i for i in range(16)] + vb = [step * 1000 - i for i in range(16)] + expect.append(va + vb + va) + a, b, c = dsts + with device_copy.active(copier): + device_copy.begin() + _copy(a, _host(va)) + _copy(b, _host(vb)) + _copy(c, a) + if before_upload is not None: + before_upload(step) + device_copy.flush(in_graph=True) + if before_replay is not None: + before_replay(step) + graph.replay() + hist[step].copy_(out) + torch.cuda.synchronize() + return hist.tolist(), expect + + +def test_early_copy_waits_for_the_previous_prologue_before_overwriting_the_window(): + """A slow GPU: the replay of step N (and its prologue) is held back behind a long kernel while the host already + uploads step N+1; that upload must wait for step N's prologue to consume the window.""" + copier, graph, dsts, out = _graph_case(early=True) + hist, expect = _history_run( + copier, + graph, + dsts, + out, + 4, + before_replay=lambda step: torch.cuda._sleep(_SLEEP_CYCLES // 4), + ) + assert hist == expect + + +def test_early_copy_prologue_waits_for_a_slow_upload(): + """A slow copy stream: each step's upload is held back behind a long kernel on the copy stream; the prologue must + wait for it (staged_ready), and the next host fill must wait for the upload to have read the window.""" + copier, graph, dsts, out = _graph_case(early=True) + + def slow_copy(step): + with torch.cuda.stream(copier._pack.copy_stream): + torch.cuda._sleep(_SLEEP_CYCLES // 4) + + hist, expect = _history_run(copier, graph, dsts, out, 4, before_upload=slow_copy) + assert hist == expect + + +def test_early_copy_with_a_slow_host(): + """A slow host: the GPU idles between steps (the window lands long before the replay).""" + copier, graph, dsts, out = _graph_case(early=True) + hist, expect = _history_run( + copier, graph, dsts, out, 3, before_replay=lambda step: time.sleep(0.05) + ) + assert hist == expect + + +@pytest.mark.parametrize("early", [False, True], ids=["prologue_copy", "early_copy"]) +def test_in_graph_replay_after_quiesce_reapplies_nothing(early): + copier, graph, dsts, out = _graph_case(early) + _step(copier, graph, dsts, (list(range(16)), list(range(16)))) + torch.cuda.synchronize() + dsts[0].fill_(7) + copier.quiesce_graph_window() + graph.replay() + torch.cuda.synchronize() + assert dsts[0].tolist() == [7] * 16 + + +def test_prologue_op_needs_early_copy_and_is_off_by_default(): + def op(window, epoch, max_desc): + raise AssertionError("not called") + + assert InGraph().prologue_op is None + assert not InGraph().early_copy + with pytest.raises(ValueError, match="needs early_copy"): + InGraph(prologue_op=op) + + +def test_in_graph_needs_parallel_scatter_and_is_off_by_default(): + assert CopyPlan().final_flush is None + assert _copier()._pack.in_graph is None + with pytest.raises(ValueError, match="needs parallel_scatter"): + CopyPlan(merge={_Site.H2D}, final_flush=InGraph()) + with pytest.raises(ValueError, match="must merge some"): + CopyPlan(parallel_scatter=True, final_flush=InGraph()) + + +def test_a_copier_without_the_in_graph_flush_captures_no_prologue(): + copier = _copier(parallel_scatter=True) + graph = torch.cuda.CUDAGraph() + with device_copy.active(copier): + torch.cuda.synchronize() + with torch.cuda.graph(graph): + assert device_copy.capture_prologue() is None + assert device_copy.capture_prologue() is None + + +def test_in_graph_replay_outside_the_pack_context_reapplies_nothing(): + copier, graph, dsts, out = _graph_case() + _step(copier, graph, dsts, (list(range(16)), list(range(16)))) + torch.cuda.synchronize() + dsts[0].fill_(7) + # What the graph runner does for a replay outside the copier's context. + assert not device_copy.is_active(copier) + copier.quiesce_graph_window() + graph.replay() + torch.cuda.synchronize() + assert dsts[0].tolist() == [7] * 16 + + +def test_step_epoch_is_off_by_default_and_needs_in_graph(): + assert not InGraph().step_epoch + copier = _copier(parallel_scatter=True, final_flush=InGraph()) + assert copier.epoch is None + with device_copy.active(copier): + assert device_copy.step_epoch() is None + assert device_copy.step_epoch() is None + assert _copier(parallel_scatter=True).epoch is None + + +@pytest.mark.parametrize("early", [False, True], ids=["prologue_copy", "early_copy"]) +def test_step_epoch_advances_once_per_replay_and_never_resets(early): + copier, graph, dsts, out = _graph_case(early, epoch=True) + with device_copy.active(copier): + assert device_copy.step_epoch() is copier.epoch + torch.cuda.synchronize() + start = int(copier.epoch.item()) # the warm-up compile launched the prologue once + assert start == 1 + for step in range(5): + va = [step * 100 + i for i in range(16)] + vb = [step * 1000 - i for i in range(16)] + # A replay with a new pack, one with an eagerly issued pack, and one with nothing to scatter. + _step(copier, graph, dsts, (va, vb), overlap=step == 2, flush=step != 3) + torch.cuda.synchronize() + assert int(copier.epoch.item()) == start + step + 1, step + # The copies are unchanged by the epoch. + _step(copier, graph, dsts, (list(range(16)), list(range(16, 32)))) + torch.cuda.synchronize() + assert out.tolist() == list(range(16)) + list(range(16, 32)) + list(range(16)) + assert int(copier.epoch.item()) == start + 6 + + +def test_step_epoch_is_ordered_before_the_graph_body(): + """A kernel after the prologue in the same graph reads the bumped epoch of its own replay.""" + copier = _copier(parallel_scatter=True, final_flush=InGraph(step_epoch=True)) + seen = torch.zeros(4, dtype=torch.int64, device="cuda") + slot = torch.zeros(1, dtype=torch.int64, device="cuda") + graph = torch.cuda.CUDAGraph() + with device_copy.active(copier): + torch.cuda.synchronize() + with torch.cuda.graph(graph): + device_copy.capture_prologue() + seen.index_copy_(0, slot, copier.epoch) + slot.add_(1) + torch.cuda.synchronize() + start = int(copier.epoch.item()) + for _ in range(4): + with device_copy.active(copier): + copier.quiesce_graph_window() + graph.replay() + torch.cuda.synchronize() + assert seen.tolist() == [start + 1, start + 2, start + 3, start + 4] + + +def test_eager_flush_does_not_advance_the_step_epoch(): + copier = _copier(parallel_scatter=True, final_flush=InGraph(step_epoch=True)) + torch.cuda.synchronize() + start = int(copier.epoch.item()) + dst = _zeros(4) + with device_copy.active(copier): + device_copy.begin() + _copy(dst, _host([1, 2, 3, 4])) + device_copy.flush() + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3, 4] + assert int(copier.epoch.item()) == start + + +@pytest.mark.parametrize("early", [False, True]) +def test_graph_window_keeps_temporary_device_sources_until_replay(early): + copier, graph, (a, b, c), out = _graph_case(early) + source = torch.arange(16, device="cuda", dtype=a.dtype) + source_ref = weakref.ref(source) + with device_copy.active(copier): + device_copy.begin() + _copy(a, source) + del source + device_copy.flush(in_graph=True) + assert source_ref() is not None + overwrite = torch.full((16,), -7, device="cuda", dtype=a.dtype) + graph.replay() + copier.on_graph_replay() + torch.cuda.synchronize() + assert out[:16].tolist() == list(range(16)) + assert overwrite.eq(-7).all() + assert source_ref() is None + + +@pytest.mark.parametrize("early", [False, True]) +def test_replay_with_a_different_copier_issues_its_copies_eagerly(early): + captured, graph, (a, b, c), out = _graph_case(early) + _step(captured, graph, (a, b, c), (list(range(16)), list(range(16)))) + active = _copier(parallel_scatter=True, final_flush=InGraph(early_copy=early)) + key = KeyType(batch_size=16, draft_len=0) + metadata = object() + runner = object.__new__(CUDAGraphRunner) + runner.graph_metadata = {key: {"attn_metadata": metadata, "spec_metadata": None}} + runner.shared_static_tensors = {"input_ids": a, "position_ids": b.view(1, 16)} + runner.config = SimpleNamespace(use_mrope=False, stage_decode_inputs=False) + runner._check_strict_buffer_stability = lambda meta: None + runner._get_num_tokens_for_key = lambda key: 16 + runner._prologue_copiers = {key: captured} + runner._step_input_prologue_keys = {} + runner._pre_replay_hooks = [] + runner._hostfunc_graph_keys = set() + runner.graphs = {key: graph} + runner.graph_outputs = {key: out} + inputs = { + "attn_metadata": metadata, + "input_ids": torch.full_like(a, 77), + "position_ids": torch.full_like(b.view(1, 16), 88), + } + with device_copy.active(active): + device_copy.begin() + runner.replay(key, inputs) + torch.cuda.synchronize() + assert out[:16].tolist() == [77] * 16 + assert out[16:32].tolist() == [88] * 16 diff --git a/tests/unittest/_torch/pyexecutor/test_device_copy_plan_buffer.py b/tests/unittest/_torch/pyexecutor/test_device_copy_plan_buffer.py new file mode 100644 index 000000000000..d7f72ceb3e7f --- /dev/null +++ b/tests/unittest/_torch/pyexecutor/test_device_copy_plan_buffer.py @@ -0,0 +1,428 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import importlib.util +import sys +from pathlib import Path + +import pytest +import torch + +# Loaded by path: the module needs only torch and tensorrt_llm._utils, not the PyTorch backend's package. +_MODULE_PATH = ( + Path(__file__).resolve().parents[4] + / "tensorrt_llm" + / "_torch" + / "pyexecutor" + / "device_copy.py" +) + + +def _load(name: str): + spec = importlib.util.spec_from_file_location(name, _MODULE_PATH) + module = importlib.util.module_from_spec(spec) + # Registered first: its dataclasses resolve their annotations through sys.modules. + sys.modules[name] = module + spec.loader.exec_module(module) + return module + + +device_copy = _load("_device_copy_under_test") + + +class _Site(device_copy.CopySiteEnum): + UPLOAD = ("test.upload", device_copy.Direction.H2D) + OTHER = ("test.other", device_copy.Direction.H2D) + TAIL = ("test.tail", device_copy.Direction.FILL) + + +def DeviceCopyBuffer(tensor: torch.Tensor, **kwargs): + return device_copy.DeviceCopyBuffer(tensor, site=kwargs.pop("site", _Site.UPLOAD), **kwargs) + + +pytestmark = pytest.mark.cpu_only + +SENTINEL = -7 + + +@pytest.fixture(autouse=True) +def _no_skipped_copy_check(monkeypatch) -> None: + monkeypatch.setattr(device_copy, "CHECK_MODE", False) + + +def _host(values: list[int]) -> torch.Tensor: + return torch.tensor(values, dtype=torch.int32) + + +def _tamper(dst: torch.Tensor) -> None: + dst.fill_(SENTINEL) + + +def test_first_call_copies() -> None: + dst = torch.zeros(8, dtype=torch.int32) + buffer = DeviceCopyBuffer(dst) + buffer.update(_host([3, 1, 2])) + assert dst.tolist() == [3, 1, 2, 0, 0, 0, 0, 0] + + +def test_a_host_buffer_never_queries_cuda(monkeypatch) -> None: + def no_cuda() -> bool: + raise AssertionError("a host-only DeviceCopyBuffer queried CUDA") + + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", no_cuda) + dst = torch.zeros(8, dtype=torch.int32) + buffer = DeviceCopyBuffer(dst) + buffer.update(_host([3, 1, 2])) + buffer.update(_host([3, 1, 2])) + buffer.fill_tail(_Site.TAIL, 3, 0) + buffer.fill_tail(_Site.TAIL, 3, 0) + assert dst.tolist() == [3, 1, 2, 0, 0, 0, 0, 0] + + +def test_unchanged_values_skip_the_copy() -> None: + dst = torch.zeros(8, dtype=torch.int32) + buffer = DeviceCopyBuffer(dst) + buffer.update(_host([3, 1, 2])) + _tamper(dst) + buffer.update(_host([3, 1, 2])) + assert dst.eq(SENTINEL).all() + + +def test_changed_values_copy() -> None: + dst = torch.zeros(8, dtype=torch.int32) + buffer = DeviceCopyBuffer(dst) + buffer.update(_host([3, 1, 2])) + _tamper(dst) + buffer.update(_host([3, 1, 4])) + assert dst[:3].tolist() == [3, 1, 4] + assert dst[3:].eq(SENTINEL).all(), "only the leading elements are written" + + +def test_prefix_of_the_last_values_skips_the_copy() -> None: + dst = torch.zeros(8, dtype=torch.int32) + buffer = DeviceCopyBuffer(dst) + buffer.update(_host([5, 6, 7, 8])) + _tamper(dst) + buffer.update(_host([5, 6])) + assert dst.eq(SENTINEL).all() + + +def test_longer_values_copy_and_extend_what_is_kept() -> None: + dst = torch.zeros(8, dtype=torch.int32) + buffer = DeviceCopyBuffer(dst) + buffer.update(_host([5, 6])) + buffer.update(_host([5, 6, 7, 8])) + assert dst[:4].tolist() == [5, 6, 7, 8] + _tamper(dst) + buffer.update(_host([5, 6, 7, 8])) + assert dst.eq(SENTINEL).all() + + +def test_shorter_changed_values_keep_the_tail() -> None: + dst = torch.zeros(8, dtype=torch.int32) + buffer = DeviceCopyBuffer(dst) + buffer.update(_host([5, 6, 7, 8])) + buffer.update(_host([9, 9])) + assert dst[:4].tolist() == [9, 9, 7, 8] + _tamper(dst) + buffer.update(_host([9, 9, 7, 8])) + assert dst.eq(SENTINEL).all() + buffer.update(_host([9, 9, 7, 1])) + assert dst[:4].tolist() == [9, 9, 7, 1] + + +def test_host_values_may_be_reused_right_away() -> None: + dst = torch.zeros(4, dtype=torch.int32) + buffer = DeviceCopyBuffer(dst) + host = _host([1, 2, 3]) + buffer.update(host) + host.fill_(0) + assert dst[:3].tolist() == [1, 2, 3] + buffer.update(_host([1, 2, 3])) + assert dst[:3].tolist() == [1, 2, 3] + + +def test_a_new_buffer_starts_over() -> None: + first = torch.zeros(4, dtype=torch.int32) + first_buffer = DeviceCopyBuffer(first) + first_buffer.update(_host([1, 2])) + second = torch.full((4,), SENTINEL, dtype=torch.int32) + second_buffer = DeviceCopyBuffer(second) + second_buffer.update(_host([1, 2])) + assert second[:2].tolist() == [1, 2] + + +def test_two_dimensional_values_are_flattened() -> None: + dst = torch.zeros(6, dtype=torch.int32) + buffer = DeviceCopyBuffer(dst) + buffer.update(torch.tensor([[1, 2], [3, 4]], dtype=torch.int32)) + assert dst[:4].tolist() == [1, 2, 3, 4] + _tamper(dst) + buffer.update(_host([1, 2, 3, 4])) + assert dst.eq(SENTINEL).all() + + +def test_debug_check_passes_a_buffer_only_the_function_wrote(monkeypatch) -> None: + monkeypatch.setattr(device_copy, "CHECK_MODE", True) + dst = torch.zeros(8, dtype=torch.int64) + buffer = DeviceCopyBuffer(dst) + buffer.update(_host([5, 6, 7, 8])) + buffer.update(_host([5, 6, 7, 8])) + buffer.update(_host([5, 6])) + assert dst[:4].tolist() == [5, 6, 7, 8] + + +def test_debug_check_reports_a_write_by_anything_else(monkeypatch) -> None: + monkeypatch.setattr(device_copy, "CHECK_MODE", True) + dst = torch.zeros(8, dtype=torch.int32) + buffer = DeviceCopyBuffer(dst) + buffer.update(_host([3, 1])) + dst[:2].copy_(_host([5, 6])) + with pytest.raises( + RuntimeError, match=r"_Site\.UPLOAD was passed unchanged=True, but its destination" + ): + buffer.update(_host([3, 1])) + + +def test_two_consumers_share_changed_then_restored_copy_state(monkeypatch) -> None: + storage = torch.full((4,), SENTINEL, dtype=torch.int32) + manager = DeviceCopyBuffer(storage) + metadata = manager + manager_view = manager.tensor.view_as(storage) + metadata_view = metadata.tensor[:2] + address = storage.data_ptr() + writes = [] + original = torch.Tensor.copy_ + + def counted(dst, src, *args, **kwargs): + if dst.data_ptr() == address: + writes.append(src.tolist()) + return original(dst, src, *args, **kwargs) + + monkeypatch.setattr(torch.Tensor, "copy_", counted) + for consumer, values in ( + (manager, [1, 2]), + (metadata, [9, 2]), + (manager, [1, 2]), + ): + consumer.update(_host(values)) + assert manager_view.tolist() == values + [SENTINEL, SENTINEL] + assert metadata_view.tolist() == values + assert storage.data_ptr() == address + metadata.update(_host([1, 2])) + manager.update(_host([1, 2])) + assert writes == [[1, 2], [9, 2], [1, 2]] + + +def test_forced_writer_keeps_a_shared_alias_cache_coherent(monkeypatch) -> None: + storage = torch.zeros(2, dtype=torch.int32) + manager = DeviceCopyBuffer(storage) + metadata = manager + writes = [] + original = torch.Tensor.copy_ + + def counted(dst, src, *args, **kwargs): + if dst.data_ptr() == storage.data_ptr(): + writes.append(src.tolist()) + return original(dst, src, *args, **kwargs) + + monkeypatch.setattr(torch.Tensor, "copy_", counted) + manager.update(_host([1, 2]), force=True) + manager.update(_host([1, 2]), force=True) + metadata.update(_host([9, 2])) + manager.update(_host([1, 2]), force=True) + metadata.update(_host([1, 2])) + assert storage.tolist() == [1, 2] + assert writes == [[1, 2], [1, 2], [9, 2], [1, 2]] + + +def test_specialized_copy_records_host_inputs_and_layout_key() -> None: + buffer = DeviceCopyBuffer(torch.zeros(4, dtype=torch.int32)) + host = _host([1, 2, 3, 4]).view(2, 2) + key = ((7, 9), 1, 2, 8) + assert not buffer.matches(host, key=key, exact=True) + buffer.record(host, key=key) + assert buffer.matches(host, key=key, exact=True) + assert not buffer.matches(host, key=((9, 7), 1, 2, 8), exact=True) + assert not buffer.matches(host.flatten(), key=key, exact=True) + assert not buffer.matches(host.to(torch.int64), key=key, exact=True) + host[0, 0] = 99 + assert not buffer.matches(host, key=key, exact=True) + + +def test_invalidation_discards_a_recorded_copy() -> None: + buffer = DeviceCopyBuffer(torch.zeros(4, dtype=torch.int32)) + host = _host([1, 2]) + buffer.record(host) + assert buffer.matches(host) + buffer.invalidate() + assert not buffer.matches(host) + buffer.update(host) + assert buffer.tensor.tolist() == [1, 2, 0, 0] + + +def test_new_layout_does_not_retain_the_previous_prefix_tail() -> None: + buffer = DeviceCopyBuffer(torch.zeros(4, dtype=torch.int32)) + buffer.record(_host([1, 2, 3, 4]), key="first") + buffer.record(_host([1, 2]), key="second") + assert buffer.matches(_host([1, 2]), key="second") + assert not buffer.matches(_host([1, 2, 3, 4]), key="second") + + +def test_check_mode_is_one_switch(monkeypatch) -> None: + """TLLM_DEBUG_MODE=1, which verified DeviceCopyBuffer's skipped copies, turns on the same CHECK_MODE as + TLLM_DEVICE_COPY_CHECK=1.""" + + def load(): + return _load("_device_copy_env_under_test").CHECK_MODE + + monkeypatch.delenv(device_copy.CHECK_ENV, raising=False) + monkeypatch.delenv("TLLM_DEBUG_MODE", raising=False) + assert load() is False + monkeypatch.setenv("TLLM_DEBUG_MODE", "1") + assert load() is True + monkeypatch.delenv("TLLM_DEBUG_MODE") + monkeypatch.setenv(device_copy.CHECK_ENV, "1") + assert load() is True + + +def test_update_returns_whether_it_issued() -> None: + buffer = DeviceCopyBuffer(torch.zeros(4, dtype=torch.int32)) + assert buffer.update(_host([1, 2])) is True + assert buffer.update(_host([1, 2])) is False + assert buffer.update(_host([1, 2]), force=True) is True + + +def test_updates_go_through_the_buffer_site(monkeypatch) -> None: + calls = [] + original = device_copy.copy + + def spy(site, dst, src, **kwargs): + calls.append((site, bool(kwargs.get("unchanged", False)))) + return original(site, dst, src, **kwargs) + + monkeypatch.setattr(device_copy, "copy", spy) + buffer = DeviceCopyBuffer(torch.zeros(4, dtype=torch.int32), site=_Site.OTHER) + buffer.update(_host([1, 2])) + buffer.update(_host([1, 2])) + assert calls == [(_Site.OTHER, False), (_Site.OTHER, True)] + + +def test_a_buffer_needs_an_upload_site_or_none() -> None: + with pytest.raises(ValueError, match="not an h2d site"): + DeviceCopyBuffer(torch.zeros(4, dtype=torch.int32), site=_Site.TAIL) + record_only = DeviceCopyBuffer(torch.zeros(4, dtype=torch.int32), site=None) + record_only.record(_host([1, 2])) + assert record_only.matches(_host([1, 2])) + with pytest.raises(ValueError, match="record-only"): + record_only.update(_host([1, 2])) + + +def test_a_failed_copy_leaves_no_record(monkeypatch) -> None: + buffer = DeviceCopyBuffer(torch.zeros(4, dtype=torch.int32)) + buffer.update(_host([1, 2, 3, 4])) + + def fail(*args, **kwargs): + raise RuntimeError("copy failed") + + monkeypatch.setattr(device_copy, "copy", fail) + with pytest.raises(RuntimeError, match="copy failed"): + buffer.update(_host([9, 9])) + assert not buffer.matches(_host([1, 2, 3, 4])) + assert not buffer.matches(_host([9, 9])) + + +def test_a_dropped_merged_copy_invalidates_the_record(monkeypatch) -> None: + drops = [] + original = device_copy.copy + + def keep_on_drop(site, dst, src, **kwargs): + drops.append(kwargs["on_drop"]) + return original(site, dst, src, **kwargs) + + monkeypatch.setattr(device_copy, "copy", keep_on_drop) + buffer = DeviceCopyBuffer(torch.zeros(4, dtype=torch.int32)) + buffer.update(_host([1, 2])) + assert buffer.matches(_host([1, 2])) + # The pack the copy was merged into is discarded: the copy never lands. + drops[-1]() + assert not buffer.matches(_host([1, 2])) + + +def test_an_eager_copy_is_snapshotted_unless_the_owner_opts_out(monkeypatch) -> None: + sources = [] + original = torch.Tensor.copy_ + + def spy(dst, src, *args, **kwargs): + sources.append(src) + return original(dst, src, *args, **kwargs) + + host = _host([1, 2]) + monkeypatch.setattr(torch.Tensor, "copy_", spy) + DeviceCopyBuffer(torch.zeros(4, dtype=torch.int32)).update(host) + assert sources[-1] is not host and sources[-1].data_ptr() != host.data_ptr() + DeviceCopyBuffer(torch.zeros(4, dtype=torch.int32), snapshot=False).update(host) + assert sources[-1].data_ptr() == host.data_ptr() + + +def test_a_view_update_is_keyed_by_the_view() -> None: + storage = torch.zeros(3, 4, dtype=torch.int32) + buffer = DeviceCopyBuffer(storage) + rows = torch.tensor([[1, 2], [3, 4]], dtype=torch.int32) + assert buffer.update(rows, view=storage[:2, :2]) is True + assert storage[:2, :2].tolist() == [[1, 2], [3, 4]] + assert buffer.update(rows, view=storage[:2, :2]) is False + # The same values into another view of the storage are a new copy. + assert buffer.update(rows, view=storage[1:3, :2]) is True + assert storage[1:3, :2].tolist() == [[1, 2], [3, 4]] + # An exact match is required: a prefix of the recorded values is not enough for a view. + assert buffer.update(rows[:1], view=storage[1:2, :2]) is True + + +def test_a_repeated_tail_fill_is_skipped_and_a_moved_start_refills(monkeypatch) -> None: + buf = torch.zeros(6, dtype=torch.int32) + record = DeviceCopyBuffer(buf, site=None) + record.fill_tail(_Site.TAIL, 2, -1) + assert buf.tolist() == [0, 0, -1, -1, -1, -1] + buf[4].fill_(SENTINEL) + record.fill_tail(_Site.TAIL, 2, -1) + assert buf[4].item() == SENTINEL, "an identical fill is skipped" + record.fill_tail(_Site.TAIL, 3, -1) + assert buf.tolist() == [0, 0, -1, -1, -1, -1] + # A write of the whole buffer (start == len) fills nothing and drops the record. + record.fill_tail(_Site.TAIL, 6, -1) + buf.fill_(SENTINEL) + record.fill_tail(_Site.TAIL, 3, -1) + assert buf[3:].tolist() == [-1, -1, -1] + + +def test_an_update_past_the_tail_start_drops_the_tail_record() -> None: + buf = torch.zeros(6, dtype=torch.int32) + record = DeviceCopyBuffer(buf) + record.update(_host([1, 2])) + record.fill_tail(_Site.TAIL, 2, -1) + record.update(_host([1, 2, 3])) + record.fill_tail(_Site.TAIL, 2, -1) + assert buf.tolist() == [1, 2, -1, -1, -1, -1] + + +def test_check_mode_verifies_a_skipped_tail_fill(monkeypatch) -> None: + monkeypatch.setattr(device_copy, "CHECK_MODE", True) + buf = torch.zeros(4, dtype=torch.int32) + record = DeviceCopyBuffer(buf, site=None) + record.fill_tail(_Site.TAIL, 1, -1) + buf[3].fill_(SENTINEL) + with pytest.raises(RuntimeError, match=r"_Site\.TAIL was passed unchanged=True"): + record.fill_tail(_Site.TAIL, 1, -1) diff --git a/tests/unittest/_torch/pyexecutor/test_device_copy_pool_shared.py b/tests/unittest/_torch/pyexecutor/test_device_copy_pool_shared.py new file mode 100644 index 000000000000..31f424fdaf09 --- /dev/null +++ b/tests/unittest/_torch/pyexecutor/test_device_copy_pool_shared.py @@ -0,0 +1,139 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""CUDA graph metadata buffers are pooled across graph keys, so their owners never skip uploads into them. + +Every graph key's metadata takes its buffers from one pool, which hands the same block to each key that asks for the +same name, and a captured forward's device writes into a block run again at each of its replays. A record of what an +upload last wrote (``DeviceCopyBuffer``) then no longer describes the device contents, so such destinations go through +the never-skipping ``device_copy.copy`` / ``device_copy.fill_tail``; only a destination its upload alone writes (the MSA +block table) keeps a record. +""" + +import pytest +import torch + +from tensorrt_llm._torch.memory_buffer_utils import Buffers +from tensorrt_llm._torch.pyexecutor import device_copy +from tensorrt_llm._torch.pyexecutor.device_copy import CopySiteEnum, DeviceCopyBuffer, Direction + +pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required") + +NUM_SLOTS = 8 + + +class _Site(CopySiteEnum): + UPLOAD = ("test_pool_shared.upload", Direction.H2D) + TAIL = ("test_pool_shared.tail", Direction.FILL) + + +@pytest.fixture(autouse=True) +def _no_check_mode(monkeypatch): + # The skipping variants below demonstrate a stale record on purpose; check mode would raise on them. + monkeypatch.setattr(device_copy, "CHECK_MODE", False) + + +def _host(values): + return torch.tensor(values, dtype=torch.int32).pin_memory() + + +def _graph_key_views(name): + """Two graph keys' metadata views from one pool, as create_cuda_graph_metadata hands them out.""" + pool = Buffers() + large = pool.get_buffer([NUM_SLOTS], torch.int32, name, reserve_buffer=True) + small = pool.get_buffer([NUM_SLOTS], torch.int32, name, reserve_buffer=True) + assert small.data_ptr() == large.data_ptr() + return large, small + + +def _capture_live_row_write(buf, rows, values): + """A graph whose replay writes buf[:rows] on the device, as on_update_kv_lens writes the live slots.""" + slots = torch.tensor(values, dtype=torch.int32, device="cuda") + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(stream), torch.cuda.graph(graph, stream=stream): + buf[:rows].copy_(slots) + torch.cuda.current_stream().wait_stream(stream) + return graph + + +def _key_b_step(buf, live, record): + """Key B's prepare: its live slots, then -1 over the padded rows its captured producers also run. With a record, + both writes skip what the record says the buffer holds (the stale behaviour); without, they always land.""" + if record is None: + device_copy.copy(_Site.UPLOAD, buf[: len(live)], _host(live)) + device_copy.fill_tail(_Site.TAIL, buf, len(live), -1) + else: + record.update(_host(live), view=buf[: len(live)]) + record.fill_tail(_Site.TAIL, len(live), -1) + + +@pytest.mark.parametrize("skipping", [False, True], ids=["never_skipped", "skipping_record"]) +def test_two_graph_keys_over_one_block_keep_the_padded_rows_unused(skipping): + key_a, key_b = _graph_key_views("msa_out_cache_loc") + replay_a = _capture_live_row_write(key_a, 6, [10, 11, 12, 13, 14, 15]) + record = DeviceCopyBuffer(key_b, site=_Site.UPLOAD) if skipping else None + + _key_b_step(key_b, [20, 21, 22], record) + torch.cuda.synchronize() + assert key_b.tolist() == [20, 21, 22, -1, -1, -1, -1, -1] + # Key A's step: its replay writes six live slots into the shared block. + replay_a.replay() + # Key B's next step, with the same live slots. + _key_b_step(key_b, [20, 21, 22], record) + torch.cuda.synchronize() + if skipping: + # The live-row upload was skipped, so key B's live rows hold key A's live slots (the tail fill follows a + # write of the live rows through a view, which drops its record, so it lands). + assert key_b.tolist() == [10, 11, 12, -1, -1, -1, -1, -1] + else: + assert key_b.tolist() == [20, 21, 22, -1, -1, -1, -1, -1] + + +@pytest.mark.parametrize("skipping", [False, True], ids=["never_skipped", "skipping_record"]) +def test_two_graph_keys_over_one_block_keep_their_uploads(skipping): + key_a, key_b = _graph_key_views("msa_block_table") + replay_a = _capture_live_row_write(key_a, 4, [7, 7, 7, 7]) + table = [1, 2, 3, 4] + record = DeviceCopyBuffer(key_b, site=_Site.UPLOAD) + + def upload(): + if skipping: + return record.update(_host(table), view=key_b[:4]) + return device_copy.copy(_Site.UPLOAD, key_b[:4], _host(table)) + + assert upload() is True + replay_a.replay() + copied = upload() + torch.cuda.synchronize() + if skipping: + assert copied is False + assert key_b[:4].tolist() == [7, 7, 7, 7] + else: + assert copied is True + assert key_b[:4].tolist() == table + + +def test_a_write_outside_the_record_must_invalidate_it(): + """Neither the upload record nor the tail record vouches for the buffer after a write outside them.""" + buf = torch.zeros(6, dtype=torch.int32, device="cuda") + record = DeviceCopyBuffer(buf, site=_Site.UPLOAD) + values = _host([1, 2]) + assert record.update(values, view=buf[:2]) is True + record.fill_tail(_Site.TAIL, 2, -1) + device_copy.copy(_Site.UPLOAD, buf[:2], _host([5, 6])) + buf[3] = 9 + record.invalidate() + assert record.update(values, view=buf[:2]) is True + record.fill_tail(_Site.TAIL, 2, -1) + torch.cuda.synchronize() + assert buf.tolist() == [1, 2, -1, -1, -1, -1] + + +def test_never_skipped_writes_without_an_active_copier_are_plain_writes(): + assert device_copy.current() is None + buf = torch.zeros(4, dtype=torch.int32, device="cuda") + assert device_copy.copy(_Site.UPLOAD, buf[:2], _host([3, 4])) is True + device_copy.fill_tail(_Site.TAIL, buf, 2, -1) + torch.cuda.synchronize() + assert buf.tolist() == [3, 4, -1, -1] diff --git a/tests/unittest/_torch/pyexecutor/test_step_inputs.py b/tests/unittest/_torch/pyexecutor/test_step_inputs.py index 30d446ba3a5c..a7861db2977e 100644 --- a/tests/unittest/_torch/pyexecutor/test_step_inputs.py +++ b/tests/unittest/_torch/pyexecutor/test_step_inputs.py @@ -982,6 +982,7 @@ def test_replay_with_a_different_pack_issues_its_copies_eagerly(early): runner._check_strict_buffer_stability = lambda meta: None runner._get_num_tokens_for_key = lambda key: 16 runner._step_input_prologue_keys = {key: captured} + runner._prologue_copiers = {} runner._pre_replay_hooks = [] runner._hostfunc_graph_keys = set() runner.graphs = {key: graph} diff --git a/tests/unittest/conftest.py b/tests/unittest/conftest.py index 86af84701dec..98896a47dec8 100644 --- a/tests/unittest/conftest.py +++ b/tests/unittest/conftest.py @@ -450,8 +450,16 @@ def torch_empty_cache() -> None: @pytest.fixture(autouse=True) def verify_skipped_device_copies(monkeypatch) -> None: + """Verify skipped and unchanged device copies in every test. + + The copy plan's check mode (device_copy.CHECK_MODE) is turned on when the test's modules imported + device_copy at collection, so tests that never touch it do not import the PyTorch backend. + """ monkeypatch.setattr( "tensorrt_llm._device_copy._verify_skipped_device_copies", True) + module = sys.modules.get("tensorrt_llm._torch.pyexecutor.device_copy") + if module is not None: + monkeypatch.setattr(module, "CHECK_MODE", True) @pytest.fixture(scope="module")