Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 37 additions & 7 deletions .claude/skills/trtllm-model-onboard-multimodal/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,30 @@ the selected items cross modality-run boundaries.

---

## Required runtime features (P0/P1)

P0 is the minimum eligibility for a new model. A P0 opt-out needs an entry in `APPROVED_EXCEPTIONS` in the conformance suite's `known_gaps.py`, reviewed by the multimodal owners. For a P1 opt-out, add the `MultimodalFeature` to the model's `multimodal_opt_outs` with one sentence and a TRTLLM issue or nvbug ID. The engine settings determine which eligible features run. A decoder that lacks a graph mode for text needs no multimodal exception for that mode, so do not list it in `multimodal_opt_outs`. Until their runtime gates land, the suite does not hold new models to the features listed in `STRUCTURAL_ONLY` in `tiers.py`, unless the model opts out of one itself. Encoder CUDA graphs count when the tower registered with `@register_vision_encoder` declares `encoder_cuda_graph_modality`. A text-only target of a vision-language architecture has one check: its `validate_multimodal_request_data` raises on media and attached rows.

<!-- BEGIN multimodal-feature-tiers -->
| Feature | Tier | What the model must provide |
| --- | --- | --- |
| `contract` | P0 | Expose the language model and tower. Supply item order, row counts and input partitions. |
| `text_only` | P0 | With `disable_mm_encoder`, omit the tower and refuse raw media and attached rows at admission. |
| `item_scheduling` | P0 | Encode whole items. Supply exact per-item costs, profiling inputs and encoder budgets. |
| `encoder_cache` | P0 | Return whole-item encoder rows with exact row counts and stable media hashes. |
| `chunked_prefill_media` | P0 | Fuse only the media rows in the current prefill chunk. |
| `kv_reuse_media` | P0 | Supply stable media hashes and token counts for prefix reuse. |
| `llm_breakable_cuda_graph` | P0 | Preserve decoder graph support for media. Use a `DecoderModelForCausalLM`, keep encoder attention out of its layer registry and prepare extra graph inputs. |
| `llm_piecewise_cuda_graph` | P0 | Preserve the decoder's piecewise graph support. Scope compilation to the language model. |
| `uint8_media` | P0 | Accept the loader's uint8 media and apply processor normalization. |
| `disagg_p_to_d` | P0 | Preserve media hashes and export and restore mRoPE state when used. |
| `encoder_dp` | P0 | Build the tower from the encoder config so replicas can encode whole items. |
| `encoder_attention_backend` | P0 | Declare supported attention backends per tower in the model class body. |
| `encoder_tp` | P1 | Shard tower weights under the encoder mapping. |
| `encoder_cuda_graph` | P1 | Provide a graph capture adapter for the tower blocks. |
| `disagg_e_to_p` | P1 | Accept attached rows with a complete item layout. Becomes P0 when the generic handoff lands. |
<!-- END multimodal-feature-tiers -->

## Phases

### Phase 0 — Gather resources
Expand All @@ -276,6 +300,8 @@ Read `config.json`'s `architectures` and `model_type`. If a `_torch/models/model

### Phase 2 — Model wrapper

Subclass `MultimodalModelMixin` today. When the shared composite base lands, subclass it or document why the target needs to implement the protocol as an adapter.

Create `tensorrt_llm/_torch/models/modeling_{name}.py`. The default pattern below mirrors `modeling_llava_next.py` and `modeling_gemma3vl.py` — a single wrapper class that composes a multimodal encoder + an LLM resolved through `AutoModelForCausalLM.from_config(text_config)`. The `*ModelBase + *Model` Base/non-Base split in `modeling_qwen2vl.py` and `modeling_qwen3vl.py` is an implementation detail for sharing one wrapper between two variants of the same family (Qwen2-VL ↔ Qwen2.5-VL; Qwen3-VL ↔ Qwen3-VL-MoE) — keep the wrapper a single class unless you have the same multi-variant need.

```python
Expand All @@ -291,8 +317,8 @@ class {Name}VisionModel(nn.Module):
class {Name}Model(MultimodalModelMixin, PreTrainedModel):
config_class = {Name}Config
supports_encoder_cache = True
# Enable after implementing and validating the Phase 3 atomic-item contracts.
supports_mm_encoder_item_scheduling = False
# P0: implement the Phase 3 atomic-item contracts.
supports_mm_encoder_item_scheduling = True

def __init__(self, model_config: ModelConfig[PretrainedConfig], *args, **kwargs):
config = model_config.pretrained_config
Expand Down Expand Up @@ -406,11 +432,12 @@ Full-request models can leave unknown costs as `None`; grouped assembly still
requires exact output row counts. Do not implement the removed
`get_mm_encoder_item_metadata` hook: the registry rejects it at wrapper creation.

**Atomic-item scheduling (optional).** Enable
`supports_mm_encoder_item_scheduling` only when items can be encoded independently.
Supply exact positive `encoder_token_lengths`, `output_embedding_lengths`, and
item selection, and implement `get_max_mm_encoder_output_embeddings` to size
the cache for one legal iteration. Missing scheduling metadata fails admission;
**Atomic-item scheduling (P0).** Set
`supports_mm_encoder_item_scheduling = True`. A model whose items cannot be encoded
independently needs an `APPROVED_EXCEPTIONS` entry. Supply exact positive
`encoder_token_lengths`, `output_embedding_lengths`, and item selection, and
implement `get_max_mm_encoder_output_embeddings` to size the cache for one legal
iteration. Missing scheduling metadata fails admission;
it does not trigger a per-request fallback. Keep lengths and partitions on the
host: `.cpu().tolist()` can synchronize and does not belong in a forward path.

Expand Down Expand Up @@ -466,6 +493,8 @@ Inherit from a related mapper for prefix surgery — don't write a one-off trans

### Phase 5 — Tests

Run `pytest tests/unittest/_torch/multimodal/conformance -k <Class>` with the registered architecture or routed target name.

**Per-model unit test (the main one)** at `tests/unittest/_torch/modeling/test_modeling_<name>.py`. Subclass `TestModelingMultimodal` from `tests/unittest/_torch/modeling/test_modeling_multimodal.py` (an abstract `unittest.TestCase`) and implement six abstract methods: `get_model_config`, `get_trtllm_model_class`, `get_hf_model_class`, `get_weight_mapper_class`, `get_model_type`, `get_model_config_class`. The base class drives a `MultimodalScenario`-parameterized run (modality ∈ `image` / `multiple_image` / `video` / `text` / `mixture_text_image` / `audio`, with optional `use_cuda_graph` / `chunked_prefill` / `kv_cache_reuse`) — comparing TRT-LLM logits to HF reference, exercising the KV cache manager, attn metadata, mrope, fusion path, and CUDA graph capture in one harness. Override `get_scenarios` to declare which combinations apply to your model. Reference: `test_modeling_qwen3vl.py`, `test_modeling_qwen2_5vl.py`, `test_modeling_nemotron_h_multimodal.py`. Test data lives under `${LLM_MODELS_ROOT}/multimodals/test_data/`.

**Hybrid linear-attention models.** Override `_dummy_request_kwargs` to return `{"use_mrope": True}` if the model uses mRoPE (allocates the 3-D position-id buffer at dummy-request time). The base class's `init_kv_cache_manager` already dispatches on `is_qwen3_hybrid` / `is_nemotron_hybrid` to build `CppMambaHybridCacheManager` — don't override unless you need a different concrete manager. Use `PyKvCacheConfig` from `llmapi.llm_args` (Pydantic), not the C++ bindings `KvCacheConfig` — `CppMambaHybridCacheManager.__init__` reads `mamba_state_cache_interval` which only exists on the Pydantic side. **CUDA-graph capture in the harness doesn't currently address the Mamba SSM state buffer** — keep `use_cuda_graph=False` in `get_scenarios` for hybrid models until that's wired through; production CUDA-graph support is independent and unaffected.
Expand Down Expand Up @@ -527,6 +556,7 @@ Follow `CONTRIBUTING.md`. Title `[JIRA/NVBUG/None][type] description`, `git comm
## Pre-PR checklist

**Architecture & registration**
- [ ] Every P0 opt-out has an owner-reviewed `APPROVED_EXCEPTIONS` entry. Every P1 opt-out has a reason and tracking ID in `multimodal_opt_outs`.
- [ ] Decorator stack in correct order: `@support_multimodal_disaggregated` (outermost, optional) → `@register_vision_encoder` → `@register_auto_model` → `@register_input_processor` (innermost).
- [ ] Model inherits `MultimodalModelMixin`; an overridden `forward` calls `prepare_multimodal_inputs` and takes no raw MM tensor args.
- [ ] `multimodal_data_device_paths` lists every GPU-resident mm field.
Expand Down
2 changes: 2 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,8 @@
/tests/unittest/_torch/modules/mamba @NVIDIA/trt-llm-models-devs
/tests/unittest/_torch/multi_gpu_modeling @NVIDIA/trt-llm-models-devs
/tests/unittest/_torch/multimodal @NVIDIA/trt-llm-models-devs
/tests/unittest/_torch/multimodal/conformance/known_gaps.py @NVIDIA/trt-llm-models-devs
/tests/unittest/_torch/multimodal/conformance/tiers.py @NVIDIA/trt-llm-models-devs
/tests/unittest/models @NVIDIA/trt-llm-models-devs
/tests/unittest/others/test_multimodal_registry.py @NVIDIA/trt-llm-models-devs

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ file is reported to the planner for ownership; it is not silently omitted.

## Test and certify

- For a vision-language target, also run `pytest tests/unittest/_torch/multimodal/conformance -k <Class>`.
- Parametrize `op.CELLS`; use `_validating.py`'s `certifying` fixture so the
entry's own `__call__` and `reference` both run. Derive the comparison band;
bands wider than torch's dtype-aware default need wrong-variant negative
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ Climb only as far as the claim needs. Each rung says what it cannot see.

## Rules that save hours

- For a media architecture, check each routed target against the [multimodal requirements](references/features.md#6-multimodal-inputs), including text-only admission.
- Start from the in-tree test that already runs the feature (each section of
the reference names them) rather than writing a launcher, server, transport
or harness: the engine moves state and serves, the flow deploys and measures,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,8 @@ extra host sync under overlap costs throughput only, which no gate sees.

## 6. Multimodal inputs

Once the shared V2 composite lands, implement a media-serving target as `ModelingV2<Target>ForConditionalGeneration` on that base. It must meet the same P0 requirements as other multimodal models. A text-only target must reject both raw media and attached encoder rows at admission, in `validate_multimodal_request_data`. See the [multimodal onboarding tiers](../../../../../../../../.claude/skills/trtllm-model-onboard-multimodal/SKILL.md#required-runtime-features-p0p1).

**What.** An input processor prepares media, an encoder embeds it, and the
embeddings replace placeholder tokens in the decoder's input
(`docs/source/features/multi-modality.md`; onboarding hooks in
Expand Down
31 changes: 31 additions & 0 deletions docs/source/torch/adding_new_model.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3. [Step-by-Step Guide](#step-by-step-guide)
1. [Model Configuration](#model-configuration)
2. [Model Definition](#model-definition)
1. [Multimodal Models](#multimodal-models)
3. [Weight Loading](#weight-loading)
4. [Model Registration](#model-registration)
1. [Core Models](#core-models)
Expand Down Expand Up @@ -128,6 +129,36 @@ Optionally, you may replace the native PyTorch modules with our implementations

For a concrete reference, check out `tensorrt_llm/_torch/models/modeling_llama.py`.

#### Multimodal Models

`tensorrt_llm._torch.models.multimodal.SupportsMultimodal` lists what the runtime
reads from a multimodal model, and `MultimodalModelMixin` implements it. Subclass
the mixin: the executor still checks for it until it reads the protocol. The
`supports_multimodal = True` class marker declares the contract. It does not check
method signatures and does not turn on any feature.

- Declare encoder caching and item scheduling separately. Item scheduling also
needs the processor's item metadata and a runtime configuration that allows it.
- List skipped `MultimodalFeature` values in `multimodal_opt_outs`, or in
`multimodal_feature_limits` when the skip depends on the deployment. The
[conformance suite](source:tests/unittest/_torch/multimodal/conformance) checks
them against the required features in the multimodal onboarding skill.
- Declare attention backends in the `supported_encoder_attention_backends`
class attribute. Its empty default declares none.
- Keep the existing `MultimodalParams` inputs and tensor outputs. Selected items
are prepared before they are copied to the GPU, and item outputs keep the
selection order.
- `multimodal_data_device_paths` stays optional for every model. Read it through
`get_multimodal_data_device_paths()`; without it, every tensor in
`multimodal_data` moves to the device.
- Expose the existing model-owned cache through `multimodal_encoder_cache` and
`initialize_multimodal_encoder_cache()`. Item scheduling and encoder output
reuse share this cache.

`EncoderGroup` still describes the modalities that share an encoder call, and
`MultimodalEncoderMixin` still provides encoder attention metadata and CUDA
graphs. Prompt expansion and disaggregated serving keep their interfaces.

### Weight Loading

The base class `DecoderModelForCausalLM` provides a `load_weights` method that loads the weights from the checkpoint file and assigns them to the corresponding layers in the model. However, if the default method does not work for `MyModelForCausalLM`, you need to implement your own `load_weights`:
Expand Down
4 changes: 2 additions & 2 deletions jenkins/scripts/cbts/cbts/rules/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ for the overall CBTS architecture.
| `test_list.py` | `TestListRule` | `testlistonly` | `tests/integration/test_lists/test-db/*.yml` |
| `visual_gen.py` | `VisualGenRule` | `visualgenonly` | `examples/visual_gen/**`, `scripts/visualgen_eval/**`, `tensorrt_llm/_torch/visual_gen/**`, `tensorrt_llm/media/**`, `tensorrt_llm/visual_gen/**` (each excl. docs), and the exact VisualGen serve schema |
| `spec_dec.py` | `SpecDecRule` | `specdeconly` | `tensorrt_llm/_torch/speculative/**`, `tensorrt_llm/models/{eagle,medusa,redrafter}/**`, `examples/{eagle,medusa,redrafter,draft_target_model,ngram}/**`, `examples/llm-api/llm_speculative_decoding.py` (each excl. docs) |
| `modeling_v2.py` | `ModelingV2Rule` | `modelingv2only` | `tensorrt_llm/_torch/modeling_v2/**`, `cpp/tensorrt_llm/modeling_v2/**`, `cpp/cmake/modules/modeling_v2.cmake` (excl. docs); per target or family for changes under `models/` and `catalog/<category>/`, from paths and imports |
| `modeling_v2.py` | `ModelingV2Rule` | `modelingv2only` | `tensorrt_llm/_torch/modeling_v2/**`, `cpp/tensorrt_llm/modeling_v2/**`, `cpp/cmake/modules/modeling_v2.cmake` (excl. docs); per target or family for changes under `models/` and `catalog/<category>/`, from paths and imports; the multimodal conformance suite for routes, shared files and targets that import the multimodal contract |
| `agent_flow.py` | `AgentFlowRule` | `agentflowonly` | `agent-flow/**` (excl. docs and out-of-scope metadata) → the single `CPU-AgentFlow-UnitTest` stage; not test-db-driven |
| `openengine.py` | `OpenEngineRule` | `openengineonly` | `tensorrt_llm/grpc/openengine/**` (excl. docs) → the `l0_cpu` block containing `unittest/grpc/openengine/` |
| `docs.py` | `DocsRule` | `docsonly` | Outside noop paths: `docs/**` and verified Python docstring edits → `CPU-Build_Docs` plus documentation-related CPU tests; other `**/*.md` / `**/*.rst` → `CPU-Build_Docs` only |
| `out_of_scope.py` | `OutOfScopeRule` | `noop` / `testsonly` | Reviewed metadata paths, root issue-template and developer-assistant folders, `.gitignore` at any depth, conventional clang filenames under `cpp/`, verified static-check-only `pyproject.toml` edits, QA/dev lists, timing caches, microbenchmarks, and CBTS infrastructure |
| `out_of_scope.py` | `OutOfScopeRule` | `noop` / `testsonly` | Reviewed metadata paths, root issue-template and developer-assistant folders, `.gitignore` at any depth, conventional clang filenames under `cpp/`, verified static-check-only `pyproject.toml` edits, QA/dev lists, timing caches, microbenchmarks, and CBTS infrastructure; a file in `CPU_TEST_INPUTS` selects the CPU test that reads it |

## PythonFormattingRule

Expand Down
27 changes: 27 additions & 0 deletions jenkins/scripts/cbts/cbts/rules/_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,33 @@ def block_filters_for_stem(
return filters


def covering_entry_filters(
yaml_index: YAMLIndex, yaml_stem: str, test_path: str
) -> dict[tuple[str, int], dict[str, set[str]]]:
"""Keep the longest entry of each `yaml_stem` block that covers `test_path`."""
filters: dict[tuple[str, int], dict[str, set[str]]] = {}
for block in yaml_index.blocks:
if block.yaml_stem != yaml_stem:
continue
covering = [
e for e in block.tests if _target_in_filter_subtree(test_path, _entry_target(e))
]
if covering:
entry = max(covering, key=lambda e: len(_entry_target(e)))
filters[(block.yaml_stem, block.block_index)] = {_entry_target(entry): {entry}}
return filters


def merge_block_filters(
filters: dict[tuple[str, int], dict[str, set[str]]],
more: dict[tuple[str, int], dict[str, set[str]]],
) -> None:
"""Add the prefixes and identifiers of `more` to `filters`."""
for key, prefixes in more.items():
for prefix, ids in prefixes.items():
filters.setdefault(key, {}).setdefault(prefix, set()).update(ids)


def strip_noop_diff_lines(diff: str) -> str:
"""Normalize a unified diff so blank- and comment-only edits don't count.

Expand Down
Loading
Loading