Skip to content

[None][refactor] Route per-step device copies through a model-declared copy plan - #20972

Open
WeiHaocheng wants to merge 2 commits into
NVIDIA:mainfrom
WeiHaocheng:feat/m3-device-copy-plan
Open

WeiHaocheng wants to merge 2 commits into
NVIDIA:mainfrom
WeiHaocheng:feat/m3-device-copy-plan

Conversation

@WeiHaocheng

@WeiHaocheng WeiHaocheng commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Problem

Between two decode CUDA graph replays, a step's inputs and metadata reach their graph-stable buffers as a chain of tiny host-to-device and device-to-device copies. Each copy is a separate operation the next graph waits for, so the GPU idles for the whole chain. MiniMax-M3 removed this idle time in #20306 and went further for Rubin in #20623, but each optimization was wired through shared runtime code. The cost of that is not the optimizations themselves; it is how they are switched on.

Every per-target optimization adds a RuntimeHooks field and an if branch in shared files. On main (a7e88dee4a):

  • RuntimeHooks has 24 fields:
    • 10 of them only decide how copies are packed or skipped: pack_step_inputs, pack_parallel_scatter, pack_in_graph, pack_early_copy, pack_step_epoch, pack_prologue_op, skip_unchanged_h2d, h2d_changed_span, pack_spec_rng_uploads and spec_small_h2d;
    • plus MsaRuntimePolicy.skip_unchanged_block_offsets and the derived skips_unchanged_metadata_uploads;
    • 4 more configure one component: msa (a 5-field MsaRuntimePolicy), sampler_d2h_side_stream, kv_lens_update_once and skip_greedy_rng_upload. The last two are read nowhere.
  • 33 lines in 5 shared runtime files read these fields: decoder runner.py 11, decoder metadata.py 11, spec_sampler_base.py 5, _util.py 3 and speculative/interface.py 3. In addition, the MSA backend and KV cache manager have 23 lines that switch on the policy.
  • Call sites choose a mechanism themselves, through 23 pack= arguments and three helpers: copy_h2d_if_changed, copy_h2d_pool_shared and step_inputs.copy_. They appear in the graph runner, decoder inputs, TRTLLM attention, speculative metadata and MSA.
  • The flags depend on each other, and only comments say so: pack_in_graph needs pack_parallel_scatter, pack_early_copy needs pack_in_graph, and pack_prologue_op needs pack_early_copy. A wrong combination is silently downgraded.
  • Kill switches add dual paths in components: SUBPAGE_IN_GRAPH, SKIP_UNCHANGED_BLOCK_TABLE and TAIL_FILL_IN_FORWARD in MSA, and SKIP_POOL_SHARED_UPLOADS in h2d_cache.py.
  • Shared runtime code carries 10 [M3 host] log lines, in runner.py, metadata.py, _util.py and profiling.py.

Each enabled or disabled combination is a path through shared code that someone must keep working and testing, although only the sm_103 and sm_107 MiniMax-M3 targets turn any of it on.

How this PR fixes it: who decides what

  • A call site only names its site: device_copy.copy(CopySite.ATTN_KV_LENS, dst, src). It does not decide whether the copy is packed or skipped.
  • The target decides when merged copies land, in its _COPY_PLAN = CopyPlan(merge=..., flush_after=..., changed_span=..., parallel_scatter=..., final_flush=InGraph(...)). The plan is validated when it is built: flush_after ⊆ merge, InGraph needs parallel_scatter, and prologue_op needs early_copy.
  • The buffer owner decides whether a copy is needed. It passes unchanged=True itself, or uses a DeviceCopyBuffer record with the copy's site.
  • Component behaviour lives in the component:
    • MSA's decode shortcuts;
    • the attention metadata's upload skipping;
    • the speculative sampler's device slot ids and side-stream store copies.
  • RuntimeHooks keeps only executor and step-loop opt-ins and model extension points: decode_prologue, executor_events, spec_offsets_op, spec_step_fold, loop_on_execution_stream, merge_tuned_tactics_across_ranks, first_content_emit_iters, mxfp8_pinned_tactics, late_compile_detect and iteration_timing_one_event.

Consequences:

  • Turning an optimization on or off for one target changes only that target's plan. Shared runtime code needs no edits.
  • A model without a plan gets plain copies.
  • Disabling is always safe: a site left out of the plan is an eager copy that lands immediately.
  • TLLM_DEVICE_COPY_CHECK=1 turns the invariants that were only comments into runtime checks:
    • no merged destination is written before its barrier;
    • no unchanged copy is stale;
    • a reader before the replay asserts with expect_landed that its copy has landed.

Before and after (main a7e88dee4a against this PR)

main this PR
RuntimeHooks fields 24 10
…of which decide copy packing or skipping 10 (+ msa.skip_unchanged_block_offsets, skips_unchanged_metadata_uploads) 0
…of which configure one component (msa ×5 subfields, sampler_d2h_side_stream, kv_lens_update_once, skip_greedy_rng_upload) 4 0
Shared runtime lines reading those fields (decoder runner/metadata, _util.py, spec metadata/sampler) 33 in 5 files 0
pack= arguments and choice of skip helper at call sites 23 pack= across graph runner, decoder inputs, attention, spec, MSA 0; every call names a site
[M3 host] logs in shared runtime code 10 in 4 files 0; one model-neutral [device copy] plan line
Kill switches / dual paths 4 (3 MSA + SKIP_POOL_SHARED_UPLOADS) 0: MSA's in-forward paths are its fixed behaviour, and A/B is an edit to a target's plan
Copy-mechanism module lines step_inputs.py 501 + h2d_cache.py 235 + _device_copy.py 91 = 827 device_copy.py 1269
runtime_hooks.py lines 288 199

The copy mechanism itself is not smaller. It moves into one module and gains merge-up-to-barrier, flush_after, plan validation and check mode. This PR does not reduce code overall. What it changes:

  • shared runtime code and call sites become simpler;
  • that code stops growing with each target's opt-ins;
  • the remaining configuration lives in the five target files that use it.

Changing a target's optimization

Before. Turning on the in-graph pack for another target meant:

  • setting pack_step_inputs, pack_parallel_scatter, pack_in_graph, pack_early_copy and pack_prologue_op (and skip_unchanged_h2d, h2d_changed_span) in the target's RuntimeHooks, in the order the comments describe;
  • checking that every call site the target cares about passes pack=True or uses the right helper;
  • adding a RuntimeHooks field plus its reads in runner.py / metadata.py / _util.py for any new copy that should be packed (as pack_spec_rng_uploads and spec_small_h2d did).

Now.

  • In the target's file:
    _COPY_PLAN = CopyPlan(
        merge={CopySite.GRAPH_INPUT_IDS, ..., CopySite.SPEC_RNG_SEEDS, MsaCopySite.BLOCK_TABLE, ...},
        changed_span={MsaCopySite.BLOCK_TABLE},
        parallel_scatter=True,
        final_flush=InGraph(early_copy=True, prologue_op=my_prologue, step_epoch=True),
    )
    An invalid combination raises when the module is imported.
  • A copy that is not yet a site needs one line at the call (device_copy.copy(site, dst, src)) and one enum member. No runtime file changes.
  • Run once with TLLM_DEVICE_COPY_CHECK=1.

Design reference

  • Sites.
    • CopySiteEnum members are (key, direction) with direction H2D, D2D or FILL; the copier checks the source device against it.
    • Shared sites live in CopySite: graph inputs, decoder inputs, TRTLLM attention, speculative RNG and the DeviceCopyBuffer users.
    • Model sites live in their own enums: MsaCopySite, and M3SpecCopySite for the M3 spec fold's arming.
  • Plan.
    • merge: the copy is staged into the step's pack. A flush issues one upload plus one scatter, serial or, with parallel_scatter, one program per copy when order-free. A merged copy lands before the next barrier; the replay is the final barrier.
    • flush_after ⊆ merge: the pack flushes right after a call whose copy was staged. This is the mechanism for a device reader that runs before the replay. The current M3 plans need none: MSA expands the sub-page table inside the forward, so no merged destination is read before the replay.
    • changed_span ⊆ merge: a DeviceCopyBuffer.update(..., changed_span=True) stages only each row's changed span. This is not done through a barrier site.
    • final_flush=InGraph(early_copy, prologue_op, step_epoch):
      • every decode graph is captured with a prologue that scatters a 16 KiB pinned window;
      • with early_copy the window is uploaded on a side copy stream;
      • prologue_op is the model's one-launch prologue;
      • with step_epoch the prologue keeps a per-replay device counter, read through device_copy.step_epoch().
      • Only the last segment rides the window, and only if it fits and is order-free; otherwise it is issued eagerly and the window is left empty.
      • There is no prologue when the graph runner has pre-replay hooks.
  • Owner side.
    • copy(..., unchanged=True) is refused during capture and verified in check mode.
    • DeviceCopyBuffer(tensor, *, site) records only issued copies and drops its record when a merged copy is discarded (on_drop). It supports views and tail fills.
  • Check mode.
    • device_copy.CHECK_MODE is read once from TLLM_DEVICE_COPY_CHECK=1 or TLLM_DEBUG_MODE=1.
    • It poisons merged destinations and verifies them at the flush, verifies merged D2D sources and unchanged copies, and backs expect_landed at readers.
What Where
Sites, plan, copier, in-graph final flush, DeviceCopyBuffer, check mode tensorrt_llm/_torch/pyexecutor/device_copy.py
Shared call sites cuda_graph_runner.py (graph inputs, final flush, prologue capture), engine/runners/decoder/inputs.py, attention/backends/trtllm.py, speculative/interface.py (RNG)
DeviceCopyBuffer users with a site (none merged except below) eagle3.py, dflash.py, spec_sampler_base.py, Mamba managers, mamba2_metadata.py, decoder inputs.py
MSA sites and behaviour MsaCopySite in sparse/minimax_m3/common.py; msa_backend.py; cache_manager.py; kernels/msa_kv_update.py
Scatter kernel and op cpp/tensorrt_llm/kernels/deviceCopyScatter.cu, thop/deviceCopyScatterOp.cpp
Plans (none has a flush_after barrier) _COPY_PLAN in minimax_m3_nvfp4__sm_103__{tp4,tp8} (eager final flush) and minimax_m3_nvfp4__sm_107__{tp2,tp4,tp8} (InGraph(early_copy, m3_step_prologue); step_epoch and changed_span={BLOCK_TABLE} on tp4/tp8; spec RNG and M3SpecCopySite merged)

Behaviour changes

  • MSA's decode shortcuts are MSA's own in every configuration. These are the short-step identity table, host-staged prefix sums, pool views built once, block ids taken from page-index views and the one-launch KV-length update.
    • Legacy modeling_minimaxm3 with implementation="msa" now gets them: TP4 c1 TPOT is 6.10 against 6.57 ms, Eagle3 TPOT is −6%, and GSM8K is 89.7 against 89.8.
    • The cost is twice the decode CUDA graphs (short and long families) and about +0.25 GiB of non-torch memory per GPU.
  • Pooled CUDA graph metadata is never skipped. Every graph key takes the same block from one pool, and a captured forward's device writes into that block rerun at each replay, so a host record cannot prove what the block holds.
    • Prompt lengths, the MSA short table, query lengths and prefix sums, query rows and offsets, and the unused-slot fill are now always written: merged on the M3 targets, eager elsewhere.
    • Only the MSA block table keeps its record, because its upload is its only writer.
    • The sm_103 plans add ATTN_PROMPT_LENS to merge.
  • MSA's in-forward paths are fixed behaviour. MSA always expands the sub-page table inside the forward, always keeps the block table's owner-side record, and always fills the out_cache_loc tail in the forward. This matches main's defaults; the three kill switches, the prepare()-time expansion and the eager tail fill are removed.
  • The speculative sampler always keeps its slot ids in a device buffer that it re-uploads only when they change, and moves its store copies to the D2H side stream whenever a step fold awaits them.
  • DeviceCopyBuffer users:
    • during capture an update is always issued;
    • TLLM_DEBUG_MODE=1 also turns on the plan checks.

Test Coverage

All results are on GB300 (sm_103), container pytorch-26.09-…-202611020300-19783. There is no Rubin (sm_107) hardware here.

Unit tests.

  • Device copy, MSA, runtime hooks, graph runner and M3 shared tests: 1054 passed and 2 skipped. This run predates removing the eager tail-fill parameter of test_msa_metadata_clears_padded_cache_slot_tail, whose cases are deleted with that path. test_msa_backend.py and test_msa_runtime_policy.py rerun on the final head: 147 passed.
    • This includes the tests ported from [None][feat] MiniMax-M3 NVFP4 low latency decode megakernel for Rubin (sm_107) TP2 / TP4 / TP8 in modeling_v2 #20623:
      • test_device_copy_in_graph.py: parallel scatter, in-graph window, early copy, step epoch and prologue capture;
      • test_device_copy_changed_span.py;
      • test_device_copy_pool_shared.py;
      • test_rng_upload_pack.py;
      • test_kv_lens_update_once.py, now without the flag;
      • the graph runner prologue test;
      • the MSA skip-routing scan.
    • It also includes test_copy_plans.py, which checks every M3 target's plan.
    • The 7 failures also fail on main (test_rng_upload_pack and 6 test_spec_sampler_step_fold tests).
  • Other suites:
    • M3 sm_103 TP4: 1222 passed. TP8: 1061 passed.
    • Kimi: 536. Decoder engine: 881. Mamba: 2065, with the same 1 failure as main.
    • _torch/speculative: the same failures as main (missing checkpoints).
  • M3 sm_107 TP2/TP4/TP8 suites, run on sm_103:
    • the same pass, skip and fail counts as main;
    • the failures need SM107 kernels;
    • the CPU-level contract scans and step-prologue tests are ported.

End to end, MiniMax-M3 NVFP4 modeling_v2 TP4, chat GSM8K:

  • TLLM_DEVICE_COPY_CHECK=1 reports no violations.
  • Per-site merged and skipped counts are identical to main.
  • One flush segment per decode replay.
  • GSM8K is 89.462 on both this PR and main (reference 88.704).

Performance against main.

  • Setup: TP4, no spec, same node, alternating runs, 16 × 512-token requests.
  • c1:
    • ITL median 1.992 against 1.993 ms;
    • TPOT 2.049 ± 0.019 against 2.060 ± 0.052 ms.
  • c4:
    • ITL median 2.960 against 2.961 ms;
    • TPOT 3.049 ± 0.041 against 3.074 ms, excluding one main outlier round.
  • Copy-path host time: 222.7 ± 3.7 against 229.9 ± 7.3 µs per step.

The refactor costs nothing at runtime.

Not covered here:

  • sm_107 end to end (no hardware);
  • DFlash/DSpark and Mamba-hybrid end to end (no checkpoints);
  • the M3 v2 Eagle3 accuracy test, which fails on main as well.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

🤖 Generated with Claude Code

Dev Engineer Review

  • The refactor replaces StepInputPack, H2DSkipCache, and copy-related RuntimeHooks fields with model-declared CopyPlan and named CopySite values.
  • DeviceCopier now owns staging, barriers, graph prologues, changed-span copies, skipped-copy records, tail fills, and optional CHECK_MODE validation.
  • Main correctness risks are plan completeness, copy ordering, pooled graph-buffer ownership, skipped-copy validity after device writes, and source lifetime during graph replay.
  • The public scatter operator changed from step_inputs_scatter to device_copy_scatter. Verify build registration, fake-op registration, and all callers.
  • MSA KV-update dispatch moved from target-local hooks into the shared MSA backend. Verify kernel-unavailable and invalid-input fallback behavior.
  • Legacy MSA wrappers and copy modules were deleted. Verify that no external or non-MiniMax caller still imports them.
  • The diff contains 99 changed files, with 5,284 additions and 4,419 deletions.
  • Reported validation is mixed. PR_Github passed, but L0_MergeRequest_PR failed. Rubin, DFlash/DSpark, Mamba-hybrid, and M3 v2 Eagle3 end-to-end validation is not available.

QA Engineer Review

  • Added tests cover copy-plan validation, eager and staged copies, barriers, changed spans, graph replay, in-graph flushes, epochs, pooled graph buffers, check mode, MSA KV-update fallback, Mamba buffers, speculative uploads, and target-specific MiniMax-M3 plans.
  • Removed tests covered the legacy H2DSkipCache, StepInputPack, and MSA hook interfaces. Replacement coverage exists, but direct legacy API compatibility coverage was removed.
  • No tests/integration/test_lists/ file changed. Existing list entries cover some changed Kimi, Mamba, Eagle3, and MiniMax-M3 contract tests. Most new device_copy and MSA unit-test files had no matching test-db/ or qa/ entry in the inspected lists.
  • Discoverable list selectors include test_eagle3_gsm8k_and_acceptance, Kimi Mamba metadata tests, M3 block-table and n_valid contract tests, and the M3 TP4 drift test. test_gen_preemption.py appears in waives.txt.
  • Coverage is needs follow-up because the L0 merge pipeline failed, most new CUDA-gated unit tests are not listed, and several end-to-end areas remain untested.

Per-File QA Perspective

Source and documentation files

  • cpp/tensorrt_llm/kernels/minimaxM3/stepInputsScatter.cu → cpp/tensorrt_llm/kernels/deviceCopyScatter.cu: Renames the CUDA kernel and launcher. Verify symbol references and runtime dispatch.
  • cpp/tensorrt_llm/kernels/minimaxM3/stepInputsScatter.h → cpp/tensorrt_llm/kernels/deviceCopyScatter.h: Renames the exported declaration. Verify ABI consistency.
  • cpp/tensorrt_llm/kernels/minimaxM3Sm107/stepPrologue.h: Updates prologue documentation and references. Verify documentation matches the new graph path.
  • cpp/tensorrt_llm/thop/CMakeLists.txt: Adds the renamed operator source. Verify build and link coverage.
  • cpp/tensorrt_llm/thop/minimaxM3/stepInputsScatterOp.cpp → cpp/tensorrt_llm/thop/deviceCopyScatterOp.cpp: Renames the Torch operator and preserves validation. Verify schema, fake-op, and caller consistency.
  • docs/source/features/torch_compile_and_piecewise_cuda_graph.md: Documents DeviceCopyBuffer, owner updates, skipped uploads, fills, and check mode. Verify examples match runtime behavior.
  • tensorrt_llm/_device_copy.py: Removes the legacy DeviceCopyBuffer. Verify no stale imports remain.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp4/modeling.py: Adds the SM103 TP4 copy plan. Verify graph-input, attention, and MSA site membership.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp4/ops/attention/msa_kv_update.py: Removes the target-local MSA wrapper. Verify backend migration.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp4/ops/msa_kv_update.py: Removes the target-local validated wrapper. Verify invalid-input fallback coverage.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8/modeling.py: Adds the SM103 TP8 copy plan. Verify TP8 site parity with TP4.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8/ops/attention/msa_kv_update.py: Removes the target-local wrapper. Verify imports and dispatch.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8/ops/msa_kv_update.py: Removes the target-local validation wrapper. Verify no callers remain.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/modeling.py: Adds SM107 TP2 speculative and graph copy settings. Verify fold buffers and early-copy behavior.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/ops/spec/m3_step_prologue.py: Migrates prologue references to device_copy. Verify ordering and epoch handling.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp4/modeling.py: Adds SM107 TP4 copy-plan settings. Verify changed-span and prologue configuration.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp4/ops/spec/m3_step_prologue.py: Migrates prologue references. Verify graph launch behavior.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp8/modeling.py: Adds SM107 TP8 copy-plan settings. Verify TP8-specific epoch and changed-span behavior.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp8/ops/spec/m3_step_prologue.py: Migrates prologue references. Verify graph compatibility.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/ops/attention/msa_kv_update.py: Updates MSA reference documentation. Verify documentation remains accurate.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/ops/attention/msa_kv_update_hook.py: Removes the shared MSA hook. Verify all hook callers use the backend kernel.
  • tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/ops/spec/copy_sites.py: Adds FOLD_SLOTS and FOLD_CTRL. Verify both sites use H2D direction and appear in applicable plans.
  • tensorrt_llm/_torch/attention/ATTENTION_DEVELOPER_GUIDE.md: Documents unchanged-upload metadata and sparse graph thresholds. Verify the contract matches implementation.
  • tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py: Moves buffers to the new copy module and removes shortcut configuration. Verify pooled-view reuse and selective skipping.
  • tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/common.py: Adds named MSA copy sites. Verify site names and directions.
  • tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/msa_kv_update.py: Adds shared MSA dispatch and reference code. Verify dtype, layout, size guards, optional outputs, and fallback.
  • tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_backend.py: Routes MSA transfers through device_copy. Verify changed spans, tail fills, reader ordering, and short-decode selection.
  • tensorrt_llm/_torch/attention/backends/trtllm.py: Adds skips_unchanged_uploads. Verify default behavior and private metadata buffers.
  • tensorrt_llm/_torch/custom_ops/cpp_custom_ops.py: Adds the fake device_copy_scatter op. Verify import-only and CPU tests.
  • tensorrt_llm/_torch/modules/mamba/mamba2_metadata.py: Tags state-index copies. Verify site routing and pinned memory.
  • tensorrt_llm/_torch/pyexecutor/_util.py: Stops propagating removed sampler copy settings. Verify sampler defaults.
  • tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py: Routes capture and replay through device_copy. Verify copier identity, flushes, quiescing, and cleanup.
  • tensorrt_llm/_torch/pyexecutor/device_copy.py: Adds the central copy-plan implementation. Verify plan validation, eager fallback, barriers, graph prologues, descriptor limits, record invalidation, source lifetime, and CHECK_MODE.
  • tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/inputs.py: Tags gather-ID and position-ID uploads. Verify destination selection and ordering.
  • tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/metadata.py: Removes runtime-hook and MSA-policy wiring. Verify metadata initialization and speculative metadata.
  • tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py: Activates model-derived copiers and flushes or discards them around forward execution. Verify exception handling and graph replay.
  • tensorrt_llm/_torch/pyexecutor/h2d_cache.py: Removes the legacy H2D cache. Verify migration completeness.
  • tensorrt_llm/_torch/pyexecutor/kv_cache/mamba_cache_manager/legacy.py: Tags legacy Mamba buffers. Verify state-index and dummy-mask uploads.
  • tensorrt_llm/_torch/pyexecutor/kv_cache/mamba_cache_manager/mamba_cache_manager_v2.py: Tags hybrid Mamba buffers. Verify site routing.
  • tensorrt_llm/_torch/pyexecutor/profiling.py: Renames an iteration-timing label. Verify log consumers.
  • tensorrt_llm/_torch/pyexecutor/runtime_hooks.py: Removes copy, sampler, and MSA fields and adds speculative/timing fields. Verify API compatibility and defaults.
  • tensorrt_llm/_torch/pyexecutor/step_inputs.py: Removes legacy packing. Verify all deferred-copy users migrated.
  • tensorrt_llm/_torch/speculative/dflash.py: Tags DFlash buffers. Verify batch-index and batch-to-slot routing.
  • tensorrt_llm/_torch/speculative/eagle3.py: Tags Eagle3 buffers. Verify batch-index and slot-ID uploads.
  • tensorrt_llm/_torch/speculative/interface.py: Removes RNG packing flags and uses named sites. Verify request-level and token-level RNG uploads.
  • tensorrt_llm/_torch/speculative/spec_sampler_base.py: Uses DeviceCopyBuffer for slot IDs. Verify cache reuse and scatter behavior.
  • tensorrt_llm/llmapi/llm_args.py: Adds sparse threshold methods. Verify MSA, Triton, and sequence-length-aware results.

Test files

Unless noted otherwise, the inspected test-db/ and qa/ lists had no matching entry for the changed test file.

  • tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_shared_draft_layers.py: Initializes pool views before buffer tests. No matching list entry was found.
  • tests/unittest/_torch/attention/sparse/msa/test_msa_backend.py: Covers kernel dispatch, taken and declined updates, fallback, disabled dynamic lengths, and tail behavior. No matching list entry was found.
  • tests/unittest/_torch/attention/sparse/msa/test_msa_kv_update.py: Covers kernel reference parity, accepted layouts, optional outputs, and rejected strided or non-int32 inputs. No matching list entry was found.
  • tests/unittest/_torch/attention/sparse/msa/test_msa_runtime_policy.py: Covers MSA table uploads, tail fills, default skipping, and shared graph metadata. No matching list entry was found.
  • tests/unittest/_torch/attention/sparse/msa/test_msa_skip_cache_routing.py: Updates source scanning for device_copy writes and pooled buffers. No matching list entry was found.
  • tests/unittest/_torch/kernels/parallel_hw_agnostic/test_custom_ops.py: Removes the deleted step_inputs import. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/kimi_k3_vl/_kimi_k3_decode_step.py: Tags DSpark batch-to-slot buffers. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_mxfp4__sm_100__tp16_moetp4ep4/test_k3_decode_worker.py: Tags the KDA dummy-request mask. It is referenced by test-db/l0_cpu.yml.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp4/attention/_m3_attn_runtime_tp4.py: Removes explicit MSA policy setup. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp4/attention/test_modeling_v2_msa_kv_update.py: Deletes target-local hook tests. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8/attention/_m3_attn_runtime_tp8.py: Removes explicit MSA policy setup. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8/test_modeling_v2_m3_tp4_drift.py: Removes deleted MSA files from drift comparison. It is referenced by test-db/l0_cpu.yml.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/attention/_m3_attn_runtime_tp2.py: Uses default MSA preparation. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_block_table_contract.py: Updates write-site scanning for device_copy. It is referenced by test-db/l0_cpu.yml.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_n_valid_contract.py: Updates destination detection for new writers. It is referenced by test-db/l0_cpu.yml.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/attention/test_modeling_v2_msa_kv_update.py: Removes hook tests and retains operator certification and guards. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_step_prologue.py: Migrates prologue tests to CopyPlan and DeviceCopier. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp4/attention/_m3_attn_runtime_tp4.py: Uses default MSA preparation. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_n_valid_contract.py: Updates writer detection. It is referenced by test-db/l0_cpu.yml.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp4/attention/test_modeling_v2_msa_kv_update.py: Removes hook tests. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_step_prologue.py: Migrates prologue tests. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp8/attention/_m3_attn_runtime_tp8.py: Uses default MSA preparation. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_n_valid_contract.py: Updates writer detection. It is referenced by test-db/l0_cpu.yml.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp8/attention/test_modeling_v2_msa_kv_update.py: Removes hook tests. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_step_prologue.py: Migrates prologue tests. No matching list entry was found.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/test_copy_plans.py: Adds plan reconstruction and target-specific plan assertions. No matching list entry was found.
  • tests/unittest/_torch/modules/kimi_kda/test_kimi_k3_mamba_metadata.py: Tags Mamba state-index buffers. It is referenced by test-db/l0_b200.yml and test-db/l0_gb300.yml.
  • tests/unittest/_torch/modules/kimi_kda/test_kimi_kda_verify_parity.py: Tags Mamba state-index buffers. It is referenced by test-db/l0_b200.yml and test-db/l0_gb300.yml.
  • tests/unittest/_torch/modules/mamba/test_mamba2_metadata.py: Covers named state-index sites and skipped-copy corruption checks. No matching list entry was found.
  • tests/unittest/_torch/pyexecutor/engine/test_decoder_forward_scopes.py: Covers copier scope, no-copier behavior, exception discard, and record invalidation. No matching list entry was found.
  • tests/unittest/_torch/pyexecutor/engine/test_decoder_runtime_hooks.py: Covers sparse threshold selection after hook removal. No matching list entry was found.
  • tests/unittest/_torch/pyexecutor/kv_cache/test_mamba_cache_manager.py: Covers named Mamba sites and skipped-copy checks. It is referenced by test-db/l0_b200.yml.
  • tests/unittest/_torch/pyexecutor/kv_cache/test_mamba_model_managers.py: Covers Mamba buffer site assignment. No matching list entry was found.
  • tests/unittest/_torch/pyexecutor/test_cuda_graph_runner.py: Covers replay with active and captured copiers. No matching list entry was found.
  • tests/unittest/_torch/pyexecutor/test_device_copy.py: Adds plan, ordering, barrier, discard, graph, check-mode, and callback tests. No matching list entry was found.
  • tests/unittest/_torch/pyexecutor/test_device_copy_buffer.py: Covers records, snapshots, site validation, failures, and tail fills. No matching list entry was found.
  • tests/unittest/_torch/pyexecutor/test_h2d_changed_span.py → tests/unittest/_torch/pyexecutor/test_device_copy_changed_span.py: Migrates changed-span tests and adds eager, discard, and barrier coverage. No matching list entry was found.
  • tests/unittest/_torch/pyexecutor/test_device_copy_in_graph.py: Covers parallel scatter, in-graph flushes, epochs, replay, and source lifetime. No matching list entry was found.
  • tests/unittest/_torch/pyexecutor/test_device_copy_pool_shared.py: Covers pooled graph buffers, stale records, invalidation, and unconditional copies. No matching list entry was found.
  • tests/unittest/_torch/pyexecutor/test_gen_preemption.py: Tags DFlash buffers in preemption fixtures. It appears in test_lists/waives.txt.
  • tests/unittest/_torch/pyexecutor/test_h2d_cache.py: Removes legacy H2D-cache tests. No matching list entry was found.
  • tests/unittest/_torch/pyexecutor/test_kv_lens_update_once.py: Removes comparisons of the deleted kv_lens_update_once hook option. No matching list entry was found.
  • tests/unittest/_torch/pyexecutor/test_runtime_hooks.py: Updates defaults, valid fields, invalid declarations, and immutability checks. No matching list entry was found.
  • tests/unittest/_torch/pyexecutor/test_step_inputs.py: Removes legacy step-input packing tests. No matching list entry was found.
  • tests/unittest/_torch/speculative/hw_agnostic/test_dflash_copy_skip.py: Tags DFlash copy buffers. No matching list entry was found.
  • tests/unittest/_torch/speculative/hw_agnostic/test_dflash_dummy_slot.py: Tags DFlash dummy-slot buffers. No matching list entry was found.
  • tests/unittest/_torch/speculative/hw_agnostic/test_dflash_worker.py: Tags DFlash worker buffers. No matching list entry was found.
  • tests/unittest/_torch/speculative/hw_agnostic/test_dspark_model_engine_runtime.py: Tags overlap-offset buffers. No matching list entry was found.
  • tests/unittest/_torch/speculative/test_eagle3.py: Tags Eagle3 batch-index and slot-ID buffers. It is referenced by test-db/l0_b200.yml and test-db/l0_h100.yml.
  • tests/unittest/_torch/speculative/test_rng_upload_pack.py: Migrates RNG upload tests to DeviceCopier plans. No matching list entry was found.
  • tests/unittest/_torch/speculative/test_spec_sampler_step_fold.py: Removes tests for deleted sampler side-stream and static-slot flags. No matching list entry was found.
  • tests/unittest/conftest.py: Routes skipped-copy verification through device_copy.CHECK_MODE. It is referenced by test-db/l0_cpu.yml.

@WeiHaocheng WeiHaocheng self-assigned this Oct 26, 2026
@WeiHaocheng WeiHaocheng changed the title Feat/m3 device copy plan [None][refactor] Route per-step device copies through a model-declared copy plan Oct 26, 2026
@WeiHaocheng WeiHaocheng changed the title [None][refactor] Route per-step device copies through a model-declared copy plan [None][refactor] Route per-step device copies through a model-declared copy plan Oct 26, 2026
@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • Resume reviews
  • Trigger review

Walkthrough

This pull request replaces legacy step-input and H2D-cache copy paths with a new device_copy system, moves MiniMax M3 copy configuration into model copy plans, relocates the MSA KV-update wrapper, and renames the scatter kernel and Torch operator to device_copy_scatter.

Changes

Device-copy and MiniMax M3 migration

Layer / File(s) Summary
Copy staging and scatter foundation
tensorrt_llm/_torch/pyexecutor/device_copy.py, cpp/tensorrt_llm/kernels/deviceCopyScatter.*, cpp/tensorrt_llm/thop/deviceCopyScatterOp.cpp, cpp/tensorrt_llm/thop/CMakeLists.txt, tensorrt_llm/_torch/custom_ops/cpp_custom_ops.py, tests/unittest/_torch/pyexecutor/test_device_copy*.py, tests/unittest/_torch/pyexecutor/test_h2d_cache.py, tests/unittest/_torch/pyexecutor/test_step_inputs.py
Adds the device_copy execution module with CopyPlan, DeviceCopier, DeviceCopyBuffer, staging, barriers, in-graph flushing, unchanged-copy checks, and changed-span uploads. Removes _device_copy, h2d_cache, and step_inputs. Renames the scatter kernel and Torch operator to device_copy_scatter.
Decoder and caller migration
tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/*, tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py, tensorrt_llm/_torch/attention/backends/trtllm.py, tensorrt_llm/_torch/speculative/*, tensorrt_llm/_torch/modules/mamba/*, tensorrt_llm/_torch/pyexecutor/kv_cache/mamba_cache_manager/*, tests/unittest/_torch/...
Decoder execution now activates a DeviceCopier from the model copy plan and uses device_copy during graph capture, replay, and cleanup. Attention, speculative, Mamba, Eagle3, DFlash, and related tests now import the new module and assign explicit CopySite values to buffers and uploads.
MiniMax M3 copy plans and speculative fold
tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/.../modeling.py, .../ops/spec/m3_step_prologue.py, .../ops/spec/copy_sites.py, tests/unittest/_torch/modeling_v2/models/minimax_m3/test_copy_plans.py, tests/unittest/_torch/modeling_v2/models/minimax_m3/.../spec/*, tests/unittest/_torch/modeling_v2/models/minimax_m3/.../attention/*
MiniMax M3 targets now expose copy_plan class attributes. SM103 plans merge shared graph, attention, and MSA copy sites. SM107 plans also merge speculative-fold and RNG sites, and may enable parallel scatter, changed-span block-table uploads, and in-graph final flushes. Prologue references and tests now use device_copy.
MSA kernel and metadata path
tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/*, tensorrt_llm/_torch/pyexecutor/runtime_hooks.py, tensorrt_llm/llmapi/llm_args.py, docs/source/features/torch_compile_and_piecewise_cuda_graph.md, tensorrt_llm/_torch/attention/ATTENTION_DEVELOPER_GUIDE.md, tests/unittest/_torch/attention/sparse/msa/*
Adds the sparse MiniMax M3 kernels/msa_kv_update.py wrapper and reference, and removes the older experimental wrappers and MSA runtime-policy hook path. MSA metadata now uses named copy sites, per-storage block-table tracking, device_copy uploads, direct kernel dispatch with fallback, and config-owned short/long graph thresholds. Docs and tests are updated to the new behavior.

Estimated code review effort: 5 (Critical) | ~120 minutes

Change: Refactor

Sequence Diagram(s)

sequenceDiagram
  participant DecoderRunner
  participant DeviceCopier
  participant AttentionMetadata
  participant device_copy_scatter
  DecoderRunner->>DeviceCopier: activate copy plan
  AttentionMetadata->>DeviceCopier: submit site-tagged copies
  DecoderRunner->>DeviceCopier: flush before replay or after graph step
  DeviceCopier->>device_copy_scatter: launch merged staged copies
Loading

Possibly related issues

Suggested reviewers: peihu-nv, vsabavat

Merge Risk: Low · previous review 37bd0

The refactor looks mergeable with minor follow-ups. A few test-hardening gaps and edge cases in the device-copy changed-span path remain open. They are unlikely to affect common workflows but should be confirmed fixed or accepted by the owners.

Security architecture risk: Low · up to 37bd0

The inspected paths retain explicit buffer ownership and ordering controls, with no demonstrated new unauthorized access path. Recovery after an interrupted handoff remains uncertain, so the assessment does not treat incomplete evidence as proof of safety.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The directly affected security-sensitive scope is device memory used by one engine's execution batches, including tables selecting KV-cache pages. A scheduling or ownership violation could affect co-batched request state, but the inspected evidence does not establish cross-request disclosure or a broader deployment-level compromise.

Trust Boundaries and Controls

  • observed — The plan is obtained from the model object, while copy descriptors are constructed from caller-owned tensor addresses and sizes. The native operator trusts raw destination addresses; comparison with the base shows that this authority and its caller obligations predate the rename.
  • observed — Main and draft block-offset guards compare destination geometry, request identities, and layout parameters. They invalidate records for context processing, capture, and mismatched inputs before recording a replacement submission.

Resilience and Maintainability Implications

  • observed — Ordering controls force pending copies to land before eager operations, reject incompatible source directions, and prevent unchanged-copy suppression during graph capture. Optional checking also detects stale records and explicitly instrumented reads before their barriers; it is not a universal production-time validation boundary.

Hardening Proposals

  • proposed — If execution is intended to continue after interrupted handoff or replay-enqueue failure, define an explicit recovery transition that invalidates unconsumed upload records and quiesces the graph window before reuse. Alternatively, make engine recreation the required terminal recovery state. This is a conditional hardening proposal, not a verified attacker-triggered defect.
Pre-merge checks: Passed 4 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Docstring Coverage Inconclusive Docstring coverage is 48.13% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed

Check name Status Explanation
Title check Passed The title clearly and concisely describes the primary change: routing per-step device copies through model-declared copy plans.
Description check Passed The description includes the required Description, Test Coverage, and PR Checklist sections. It explains the problem, solution, behavior changes, validation results, and known test limitations.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 48.13% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 214 functions across 70 files. (1 skipped: 1 too large.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85511 [ run ] completed with state SUCCESS. Commit: c16477b
/LLM/main/L0_MergeRequest_PR pipeline #70780 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@WeiHaocheng
WeiHaocheng force-pushed the feat/m3-device-copy-plan branch 2 times, most recently from 090bf6e to 6fbd8ba Compare November 18, 2026 03:01
@WeiHaocheng

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85645 [ run ] triggered by Bot. Commit: 6fbd8ba Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85645 [ run ] completed with state SUCCESS. Commit: 6fbd8ba
/LLM/main/L0_MergeRequest_PR pipeline #70904 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@WeiHaocheng

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85782 [ run ] triggered by Bot. Commit: 2a005b4 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85782 [ run ] completed with state SUCCESS. Commit: 2a005b4
/LLM/main/L0_MergeRequest_PR pipeline #71028 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@WeiHaocheng

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86048 [ run ] triggered by Bot. Commit: fc4fdd7 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86048 [ run ] completed with state ABORTED. Commit: fc4fdd7

Link to invocation

A model may declare a CopyPlan (copy_plan) naming the copy sites whose
per-step copies are staged into one pack and issued as one upload and
one scatter launch, up to plan-declared barriers and at the latest at
the decode graph replay, optionally by the graph's own prologue
(device_copy.py). Buffer owners skip their unchanged uploads with
device_copy.DeviceCopyBuffer; TLLM_DEVICE_COPY_CHECK=1 verifies both.

The plan is off by default and coexists with the step-input pack:
* An engine has a copier only when its model declares a plan; the
  runner and CUDA graph runner open, flush, capture and discard it
  beside the step-input pack.
* The shared sites that gain a CopySite (graph static inputs, decoder
  position ids, attention prompt/KV lengths, speculative RNG uploads)
  go through the plan only while a copier is active and otherwise keep
  their original path, so a model without a plan is unchanged.
* A model that declares a plan and also turns on the step-input pack
  or the skip cache is refused (check_one_copy_path).
* The pack is issued with the existing trtllm::step_inputs_scatter op.

Tests cover the plan, the in-graph final flush, changed-span and
pool-shared uploads, the owner-side record, and the coexistence.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
A developer guide page describes the copy plan, how a model adopts it,
and how it coexists with the step-input pack until every model has
moved. AGENTS.md points new per-step copies at device_copy and asks
for the same additive approach whenever a shared runtime mechanism is
replaced: add the new one beside the old, opt-in and off by default,
move models in follow-ups, and remove the old one last.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
@WeiHaocheng
WeiHaocheng force-pushed the feat/m3-device-copy-plan branch from fc4fdd7 to 6351166 Compare November 20, 2026 16:19
@WeiHaocheng
WeiHaocheng requested a review from a team as a code owner November 20, 2026 16:19
@WeiHaocheng
WeiHaocheng requested a review from hyukn November 20, 2026 16:19
@WeiHaocheng

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86526 [ run ] triggered by Bot. Commit: 6351166 Link to invocation

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-compatible Accepted LLM API contract change that is backwards-compatible

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants