Skip to content
12 changes: 12 additions & 0 deletions docs/source/torch/adding_new_model.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,18 @@ Optionally, you may replace the native PyTorch modules with our implementations

For a concrete reference, check out `tensorrt_llm/_torch/models/modeling_llama.py`.

#### Multimodal Models

A Modeling V2 vision-language model has the composite shape of a Modeling V1
one: it holds the Modeling V2 language model as `llm` and the tower as
`mm_encoder`, and `MultimodalModelMixin` runs it. Name it
`ModelingV2<Target>ForConditionalGeneration` (never `...VLM` or `...Model`) and
subclass `ModelingV2ForConditionalGeneration`, the shared base. Declare only
what differs from the base; code that several targets would copy goes into the
base. The language model target keeps its name, `ModelingV2<Target>`. See
"Multimodal targets" in `tensorrt_llm/_torch/modeling_v2/README.md` for the
template.

### Weight Loading

The base class `DecoderModelForCausalLM` provides a `load_weights` method that loads the weights from the checkpoint file and assigns them to the corresponding layers in the model. However, if the default method does not work for `MyModelForCausalLM`, you need to implement your own `load_weights`:
Expand Down
98 changes: 95 additions & 3 deletions tensorrt_llm/_torch/modeling_v2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,9 @@ double underscores between the three parts:
The target class name contains the same three parts in CamelCase, for example
`ModelingV2DeepseekR10528Nvfp4Sm10xDefault` in
[`r1_0528_nvfp4__sm_10x__default`](models/deepseek_v3/r1_0528_nvfp4__sm_10x__default/).
These names are conventions, not requirements.
These names are conventions, not requirements. The one rule is for a
vision-language model, whose class name ends with `ForConditionalGeneration`
(see [Multimodal targets](#multimodal-targets)).

A target can contain several execution paths. The DeepSeek R1 target, for
example, has separate `PrefillTarget` and `DecodeTarget` classes and picks one
Expand All @@ -93,6 +95,7 @@ on each forward step. In this guide, "target" always means the whole directory.
modeling_v2/
├── _router_index.py # Route registration and target resolution
├── _native.py # Loads family native libraries
├── multimodal.py # Base class of vision-language targets
├── catalog/
│ ├── _op.py # OpWrapper: the operation contract
│ └── <category>/ # Operations used by more than one family
Expand Down Expand Up @@ -134,6 +137,11 @@ family's and its own operations and kernels. From the rest of TensorRT-LLM,
it uses only runtime interfaces, for example:

* `DecoderModel`, `DecoderModelForCausalLM`, `ModelConfig` and `Mapping`.
* `MultimodalModelMixin` and `MultimodalEncoderMixin`. A vision-language target
inherits the first through `ModelingV2ForConditionalGeneration`
([`multimodal.py`](multimodal.py)), and its tower the second. It does not
subclass or call a Modeling V1 model, vision tower, layer, attention backend
or MoE.
* Attention and speculative metadata, and the KV cache managers.

A target does not import, subclass or call Modeling V1 model code: the model
Expand Down Expand Up @@ -172,8 +180,11 @@ example that uses no Modeling V1 model code.
## Model lifecycle

A target plugs into TensorRT-LLM through the existing `DecoderModel` and
`DecoderModelForCausalLM` interfaces. It inherits these classes directly, not
a Modeling V1 model class, and can override any of these steps:
`DecoderModelForCausalLM` interfaces. A language model target inherits these
classes directly, not a Modeling V1 model class. A vision-language target
subclasses `ModelingV2ForConditionalGeneration`, which holds such a language
model target as `llm` (see [Multimodal targets](#multimodal-targets)). Either
can override any of these steps:

* **Constructor:** builds the model structure and parameters.
* **`load_weights`:** loads checkpoint weights into the model. Put weight
Expand All @@ -192,6 +203,87 @@ classes are fine when they make the code clearer or faster. The
its [weight loader](models/gpt_oss/gpt_oss_120b__sm_10x__default/weights.py)
are a small example.

## Multimodal targets

A vision-language architecture routes to one of two kinds of target. The
resolver checks every class a `route` returns: when the architecture's input
processor takes media, the class must be one of these two (or an existing
target that encodes media itself, see below), or startup fails.

**A vision-language model** runs media through the shared multimodal runtime
and has the composite shape of a Modeling V1 vision-language model: it holds a
language model target as `llm` and the encoder tower as `mm_encoder`, and
`MultimodalModelMixin` runs it. Subclass `ModelingV2ForConditionalGeneration`
from [`multimodal.py`](multimodal.py). This shared base does the common work
once: the config split, admission, encoder caching and the current chunk's rows
(through `MultimodalModelMixin`), the compile scope, the weight split, and what
the engine reads off the routed class and model. A target declares what
differs:

```python
class ModelingV2MyTargetForConditionalGeneration(ModelingV2ForConditionalGeneration):
language_model_class = ModelingV2MyTarget # the language model target
mm_encoder_class = MyTower # MyTower(vision_config, dtype)
mm_encoder_weight_prefix = "model.visual." # the tower's checkpoint keys

def encode_multimodal_inputs(self, multimodal_params):
return torch.cat([encode(self.mm_encoder, p) for p in multimodal_params])
```

`route` returns this class. The tower is an `nn.Module` and
`MultimodalEncoderMixin` that declares and loads its own weights. The base
builds it as `mm_encoder_class(vision_config, dtype)`, from the composite
config's `vision_config` at the activation dtype; for a tower built another
way, override `build_mm_encoder` instead. By default
the rows reach the language model fused into `inputs_embeds`. With
`language_model_takes_rows = True`, `input_ids` stay as they are and the
language model takes `multimodal_rows=(rows, mm_token_indices)`; the
[qwen4_exp target](models/qwen4_exp/qwen3_8_flash_next_nvfp4__sm_103__default/modeling.py)
does this because its PLE layer hashes `input_ids`. Override
`validate_media_request` to refuse, at admission, media the tower cannot
encode. To pass extra inputs to the language model, as V1 Qwen3-VL does for
DeepStack, override `get_language_model_extra_forward_kwargs`.

**A text-only target** serves the text half of the checkpoint. It defines
`validate_multimodal_request_data` and raises when
`tensorrt_llm.inputs.multimodal.has_media_or_embeddings` is true, so a request
with media fails alone at admission instead of failing its batch in `forward`.
M-RoPE positions alone, as a disaggregated generation request carries, are not
media.

**A target that encodes media itself** does not use the base. The qwen4_exp
FP8 target encodes, in its own forward, only the frames of the current prefill
chunk. It refuses at admission, in a `validate_request` of its own class (the
resolver does not count an inherited one), what its tower cannot take, and it
gets no shared encoder cache. The contract test lists such targets by name; a
new target is not added to that list, and an entry that is no longer needed
fails the test.

With `disable_mm_encoder` the model builds no tower, loads no encoder weights
and refuses media like a text-only target. With a tower, the breakable prefill
CUDA graph runs when the language model takes the rows itself
(`language_model_takes_rows`), as for the language model alone; with rows fused
into `inputs_embeds` it is refused at startup. A model with a tower does not
run its encoder on a side stream: the route declines
`encoder_side_stream_max_ahead > 0`.

**The rule for vision-language models:**

* One shared base. Code that every V2 vision-language model needs lives in
`ModelingV2ForConditionalGeneration`. A target declares only what differs;
code that several targets would copy goes into the base.
* One name. A V2 class that serves media is
`ModelingV2<Target>ForConditionalGeneration` and subclasses the base, never
`...VLM` or `...Model`. Its language model target stays `ModelingV2<Target>`.
A text-only target routed for a vision-language checkpoint keeps its name
until it serves media.

`test_modeling_v2_contracts.py` checks both kinds of target from the source, and
runs each text-only target's admission hook on media, embeddings and M-RoPE
positions. For the name, it checks that a routed multimodal model subclasses the base and that
its name ends with `ForConditionalGeneration`, and that no target class ends
with `VLM`. Review checks the `ModelingV2<Target>` part.

## Routing

Each family's `routing.py` defines one `route` function, registered with
Expand Down
70 changes: 69 additions & 1 deletion tensorrt_llm/_torch/modeling_v2/_router_index.py
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,71 @@ def _refuse_unhonored_moe_tactic(
)


def _architecture_takes_media(architecture: str) -> bool:
"""Whether requests for ``architecture`` can carry media: the input processor registered
for its built-in class, which the LLM API serves every deployment of it with, is a
multimodal one."""
from tensorrt_llm._torch.models.modeling_utils import get_registered_model_class
from tensorrt_llm.inputs.registry import INPUT_PROCESSOR_REGISTRY, BaseMultimodalInputProcessor

model_cls = get_registered_model_class(architecture)
processor_cls = INPUT_PROCESSOR_REGISTRY.get_input_processor_cls(model_cls)
return processor_cls is not None and issubclass(processor_cls, BaseMultimodalInputProcessor)


def _check_multimodal_contract(target: type, architecture: str) -> None:
"""Raise ``ValueError`` when a target of an architecture whose requests can carry media
neither runs media through the multimodal runtime nor refuses it at admission.

The first is a ``ModelingV2ForConditionalGeneration`` (``multimodal.py``), or any other
``MultimodalModelMixin`` model or class that sets the ``supports_multimodal`` marker; the
second defines ``validate_multimodal_request_data`` and raises there on media. A target that
encodes media in its own forward refuses what its tower cannot take in a ``validate_request``
of its own class; an inherited one, such as a token range check, does not count. A target
that does none of these would run placeholder tokens as text or fail a whole batch
in ``forward``: a bug in the target, so this raises in every mode (a plain ``ValueError``,
not a ``NoTarget``).
"""
from tensorrt_llm._torch.models.modeling_multimodal_mixin import MultimodalModelMixin

if (
issubclass(target, MultimodalModelMixin)
or getattr(target, "supports_multimodal", False) is True
or callable(getattr(target, "validate_multimodal_request_data", None))
or callable(vars(target).get("validate_request"))
):
return
if _architecture_takes_media(architecture):
raise ValueError(
f"{target.__qualname__} is the target for {architecture!r}, whose requests can "
"carry images, videos or audio. Make it a ModelingV2ForConditionalGeneration "
"subclass to serve media, or define validate_multimodal_request_data to refuse "
"media at admission and serve text only."
)


def _refuse_encoder_side_stream(config: "ModelConfig", target: type) -> None:
"""Raise ``NoTarget`` when a ``ModelingV2ForConditionalGeneration`` target would build its
tower and the engine runs the encoder on a side stream: the base runs its encoder inside
the step only. ``auto`` then uses the built-in implementation. An encoder-only deployment
(``mm_encoder_only``) builds the architecture's encoder model, not this one."""
from .multimodal import ModelingV2ForConditionalGeneration

if (
not issubclass(target, ModelingV2ForConditionalGeneration)
or config.disable_mm_encoder
or config.mm_encoder_only
):
return
mm_config = config.multimodal_config
if mm_config is not None and mm_config.encoder_side_stream_max_ahead > 0:
raise NoTarget(
"multimodal_config.encoder_side_stream_max_ahead="
f"{mm_config.encoder_side_stream_max_ahead}; {target.__name__} does not run its "
"encoder on a side stream"
)


def modeling_v2_resolve(config: "ModelConfig", llm_args: "TorchLlmArgs") -> Optional[type]:
"""Decide which modeling_v2 target, if any, this deployment builds.

Expand All @@ -314,7 +379,8 @@ def modeling_v2_resolve(config: "ModelConfig", llm_args: "TorchLlmArgs") -> Opti
for the architecture, or when that ``route`` raises ``NoTarget``.
In ``require`` every one of those but ``off`` raises ``ValueError`` instead.
It also returns None for a ``@disable_v2_by_default`` target when
``modeling_v2`` was left at its default.
``modeling_v2`` was left at its default. A target that breaks the multimodal
contract (:func:`_check_multimodal_contract`) raises in every mode.

"""
mode = ModelingV2Mode.of(llm_args)
Expand All @@ -334,6 +400,8 @@ def modeling_v2_resolve(config: "ModelConfig", llm_args: "TorchLlmArgs") -> Opti
target = route(config, llm_args, _device_compute_capability())
if target is not None:
_refuse_unhonored_moe_tactics(config, target, arch)
_check_multimodal_contract(target, arch)
_refuse_encoder_side_stream(config, target)
except NoTarget as why:
if mode is ModelingV2Mode.REQUIRE:
raise ValueError(f"modeling_v2='require', but no target fits {arch!r}: {why}") from None
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
from tensorrt_llm._torch.speculative import get_spec_worker, should_use_separate_draft_kv_cache
from tensorrt_llm._utils import mpi_disabled
from tensorrt_llm.functional import AllReduceFusionOp, AllReduceStrategy
from tensorrt_llm.inputs.multimodal import has_media_or_embeddings
from tensorrt_llm.logger import logger
from tensorrt_llm.mapping import Mapping as ParallelMapping

Expand All @@ -73,7 +74,6 @@

if TYPE_CHECKING:
from tensorrt_llm._torch.pyexecutor.guided_decoder import CapturableGuidedDecoder
from tensorrt_llm._torch.pyexecutor.llm_request import LlmRequest
from tensorrt_llm._torch.speculative.interface import SpecMetadata

#: The TP all-reduce of more rows than this runs on NCCL; fewer on the one-shot kernel,
Expand Down Expand Up @@ -746,15 +746,13 @@ def infer_max_seq_len(self) -> int:
"""The text config's context limit; the model has no RoPE (NoPE MLA, KDA)."""
return int(self.text_config.max_position_embeddings)

def validate_request(self, request: LlmRequest) -> None:
"""Reject media on admission, as that request's own error: the vision tower is not built,
and the forward would assert on the whole step."""
data = request.py_multimodal_data or {}
media = [kind for kind in ("image", "video") if data.get(kind)]
if media:
def validate_multimodal_request_data(self, mm_data: dict) -> None:
"""Admission check: this target serves text only, so a request with media or
embeddings fails alone here instead of failing its batch in forward. A payload of
M-RoPE positions only passes."""
if has_media_or_embeddings(mm_data):
raise ValueError(
f"this GLM-5.3-Flash deployment serves text only (disable_mm_encoder); the "
f"request carries {' and '.join(media)} input"
f"{type(self).__name__} serves text only; send no images, videos or embeddings"
)

def set_guided_decoder(self, guided_decoder: CapturableGuidedDecoder) -> bool:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@
from tensorrt_llm._torch.pyexecutor.config_utils import unwrap_glm5_next_text_config
from tensorrt_llm._torch.utils import AuxStreamType
from tensorrt_llm._utils import async_tensor_h2d
from tensorrt_llm.inputs.multimodal import has_media_or_embeddings
from tensorrt_llm.logger import logger
from tensorrt_llm.mapping import Mapping
from tensorrt_llm.models.modeling_utils import QuantConfig
Expand All @@ -84,9 +85,6 @@
# and the certification is per arch.
_SM = (10, 0)

#: ``multimodal_data`` entries that carry media for an encoder, or encoder output in place of it.
_MEDIA_KEYS = frozenset({"audio", "image", "video", "multimodal_embedding"})

#: Stock code the generic path runs outside the catalog, declared so it is not consumed silently: every
#: tensorrt_llm import of this module that computes (this target's unit tests check the list both ways, with
#: ``_glm5_next_generic_calls.py``). An entry leaves the list when a catalog entry replaces it.
Expand Down Expand Up @@ -128,6 +126,8 @@
"tensorrt_llm._torch.modules.embedding.Embedding",
"tensorrt_llm._torch.distributed.AllReduce",
"tensorrt_llm._torch.modules.multi_stream_utils.maybe_execute_in_parallel",
# The admission check that refuses media (`validate_multimodal_request_data`).
"tensorrt_llm.inputs.multimodal.has_media_or_embeddings",
)


Expand Down Expand Up @@ -2087,14 +2087,10 @@ def load_weights(self, weights: Any, weight_mapper: BaseWeightMapper | None = No
_weights.load(self, weights, weight_mapper)

def validate_multimodal_request_data(self, mm_data: dict) -> None:
"""Refuse, at admission, a request whose multimodal payload carries media: this target runs no encoder.

Refusing here fails that request alone; the forward would otherwise serve its text without the media.
A payload without media is accepted.
"""
media = sorted(key for key in _MEDIA_KEYS if mm_data.get(key) is not None)
if media:
"""Admission check: this target serves text only, so a request with media or
embeddings fails alone here instead of failing its batch in forward. A payload of
M-RoPE positions only passes."""
if has_media_or_embeddings(mm_data):
raise ValueError(
f"{type(self).__name__} serves this checkpoint's language model only and does not accept "
f"multimodal input; the request carries {', '.join(media)}"
f"{type(self).__name__} serves text only; send no images, videos or embeddings"
)
Loading