Skip to content

[TRTLLM-17780][feat] Run Modeling V2 vision-language models through the multimodal runtime (V1-style composite model + text-only guards) - #22450

Draft
yechank-nvidia wants to merge 6 commits into
NVIDIA:mainfrom
yechank-nvidia:yechank/mm-modeling-v2-wrapper
Draft

yechank-nvidia wants to merge 6 commits into
NVIDIA:mainfrom
yechank-nvidia:yechank/mm-modeling-v2-wrapper

Conversation

@yechank-nvidia

@yechank-nvidia yechank-nvidia commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

@coderabbitai summary

Description

Why. Modeling V2 routes four vision-language architectures, but none of their targets runs as a multimodal model. The qwen4_exp NVFP4 target runs its vision tower inside the language model forward, so it gets no encoder cache and no encoder memory reserve at startup, and it runs the tower again on the whole request for every prefill chunk that has media. The Kimi K3, MiniMax-M3 and GLM-5.3-Flash targets serve text only, and each one treats an image request in its own way: some fail the whole batch in forward, some check a fixed key list. This PR gives a V2 vision-language target the same composite shape V1 vision-language models have, through one shared base class, and makes every text-only target refuse media at admission with one shared helper. No new flag, env var or config.

Based on main. The new base inherits MultimodalModelMixin, so every executor check on main (isinstance(model, MultimodalModelMixin)) already accepts it. The route-time check also accepts a class with the supports_multimodal marker of #20906.

Class shape.

MultimodalModelMixin, PreTrainedModel                      (shared, V1 and V2)
  `-- ModelingV2ForConditionalGeneration                    (modeling_v2/multimodal.py, new shared base)
       llm        = language_model_class(config copy; it adopts the text config)
       mm_encoder = mm_encoder_class(vision_config, dtype), or build_mm_encoder(...);
                    None with disable_mm_encoder
       does once: config split, admission, row selection, forward hand-off,
                  weight split, compile scope, what the engine reads off the model
       `-- ModelingV2Qwen38FlashNextNvfp4Sm103DefaultForConditionalGeneration   (routed, 29 lines)
            language_model_class      = ModelingV2Qwen38FlashNextNvfp4Sm103Default
            mm_encoder_class          = VisionTower   (nn.Module, MultimodalEncoderMixin)
            mm_encoder_weight_prefix  = "model.visual."
            language_model_takes_rows = True   (input_ids kept for PLE; rows go in at mm_token_indices)

What.

  • Text-only targets (14 classes: 6 Kimi K3, 5 MiniMax-M3, 3 GLM-5.3-Flash): validate_multimodal_request_data raises when the new helper has_media_or_embeddings() is true. Only that request fails. M-RoPE-only payloads (disaggregated generation) pass. The GLM targets move their own checks to this hook and helper.
  • qwen4_exp NVFP4 honors disable_mm_encoder: no tower, no vision weights. The vision weights move to their own table, loaded by the tower.
  • Mixin: prepare_multimodal_rows() is split out of prepare_multimodal_inputs(), which now calls it and then fuses. V1 models keep the same calls in the same order.
  • New shared base ModelingV2ForConditionalGeneration. It builds llm on a config copy from language_model_class and the tower from mm_encoder_class (as mm_encoder_class(vision_config, dtype); a tower built another way overrides build_mm_encoder), forwards what the engine reads (lm_head, draft_*, spec_worker, multimodal_token_ids, class hooks), compiles only llm.model, checks the token range at admission, selects only requests with media, and splits the weights by the tower's key prefix. llm gets the engine's weight_mapper, bound to llm, when its load_weights takes one. The rows reach llm fused into inputs_embeds (default) or, with language_model_takes_rows, as (rows, mm_token_indices) beside unchanged input_ids. The base imports no routing module.
flowchart LR
  P["Step"] --> S{"Mixin: requests<br/>with media?"}
  S -->|"none"| LLM["llm.forward"]
  S -->|"some"| C{"Encoder cache /<br/>earlier chunk"}
  C -->|hit| ROWS["Encoder rows"]
  C -->|miss| ENC["mm_encoder"] --> ROWS
  ROWS -->|"default"| F["Fuse into inputs_embeds"] --> LLM
  ROWS -->|"language_model_takes_rows"| K["input_ids kept +<br/>(rows, mm_token_indices)"] --> LLM
  X["get_language_model_extra_forward_kwargs<br/>(e.g. DeepStack, M-RoPE)"] -.-> LLM
Loading
  • Route-time check: when an architecture's input processor takes media, the class its route returns must be a multimodal model (a MultimodalModelMixin model or the supports_multimodal marker), have the text-only admission hook (validate_multimodal_request_data), or encode media in its own forward and have a validate_request of its own class (an inherited one does not count; today only the qwen4_exp FP8 target, an own-encoder target that later becomes a selected_rows adapter; this PR does not change it). Otherwise startup fails with a ValueError, under auto too. Startup checks that the hook is there; the contract test below runs each text-only hook on media. The router finds the input processor through a new public accessor, INPUT_PROCESSOR_REGISTRY.get_input_processor_cls().
  • The router also declines a base subclass that builds its tower while encoder_side_stream_max_ahead > 0, so auto uses the built-in model. An encoder-only worker (mm_encoder_only) is not declined: it builds the architecture's encoder model, not this one.
  • The breakable prefill gate also accepts a composite whose llm is a decoder and that sets language_model_takes_rows, so qwen4_exp NVFP4 with a tower keeps the breakable prefill graph, as on main.
  • qwen4_exp NVFP4 routes to a 29-line subclass of the base; the tower moved there from the language model target.
  • Kimi K3 SM107 forward refuses media with the same helper as admission, so an M-RoPE-only payload passes both.
  • A contract test for every class routed for a media architecture: source checks, each text-only admission hook run on media, embeddings and M-RoPE positions (exact shared message), and a list of own-encoder targets that only shrinks. Also a test of the naming rule below, and a "Multimodal targets" section in the Modeling V2 README.
flowchart TB
  R["Modeling V2 router<br/>route() + contract check"]
  R -->|"target serves media"| VLM["ModelingV2&lt;Target&gt;ForConditionalGeneration"]
  R -->|"target encodes media itself<br/>(qwen4_exp FP8, unchanged)"| OWN["ModelingV2&lt;Target&gt;<br/>chunk-local tower in its forward,<br/>own validate_request"]
  R -->|"target serves text only"| TXT["ModelingV2&lt;Target&gt;<br/>refuses media at admission"]
  R -->|"no target fits (auto)"| V1["Modeling V1 model"]
  R -->|"class has no multimodal base<br/>and no admission hook"| ERR["Startup fails (ValueError),<br/>also under auto"]
  VLM -.->|"route declines (auto),<br/>e.g. encoder side stream"| V1
  VLM -->|subclasses| BASE["ModelingV2ForConditionalGeneration<br/>(shared base)"]
  BASE -->|inherits| MIX["MultimodalModelMixin<br/>shared runtime, same as V1"]
  VLM -->|"llm"| LLM["Modeling V2 language model target"]
  VLM -->|"mm_encoder"| ENC["Encoder tower"]
Loading

Rule for V2 vision-language models. A V2 vision-language model has the composite shape of a V1 one: the V2 language model as llm and the tower as mm_encoder, run by MultimodalModelMixin. Code that every V2 vision-language model needs lives in one shared base, ModelingV2ForConditionalGeneration; a target declares only what differs, and code that several targets would copy goes into the base. A V2 class that serves media through the shared runtime is named ModelingV2<Target>ForConditionalGeneration and subclasses the base, never ...VLM or ...Model. The qwen4_exp FP8 target, which encodes media in its own forward, is the one listed exception. Language model targets keep their ModelingV2<Target> names. Kimi K3, GLM-5.3-Flash and MiniMax-M3 serve text only and keep their names; when they get media, they follow this rule. test_modeling_v2_contracts.py checks that such a class subclasses the base and that its name ends with ForConditionalGeneration, and that no target class ends with VLM; review checks the ModelingV2<Target> part.

Since the last push. Rebased onto main (42c1287) without #20906 (34 own files before, 45 now). Main added five text-only targets (4 Kimi K3 NVFP4, 1 GLM-5.3-Flash FP8) and the qwen4_exp FP8 target; the text-only ones now use the shared admission hook, and the FP8 target is accepted as a target that encodes media itself. The side-stream decline moved from the base constructor to the router, and the base now passes weight_mapper on. qwen4_exp NVFP4 keeps breakable prefill with a tower, as on main. From review: the base takes the tower as mm_encoder_class, built as mm_encoder_class(vision_config, dtype) next to language_model_class, and build_mm_encoder stays as the override for a tower built another way; the comment on the copied language model flags now says who reads them, and a subclass that swaps in another language model takes that model's flags (or the readers' defaults), never its parent's.

Behavior changes.

  • encoder_side_stream_max_ahead > 0 with a built tower now falls back to V1 under auto (fails under require).
  • qwen4_exp NVFP4 refuses precomputed multimodal_embedding input at admission (it was ignored before).
  • qwen4_exp NVFP4 serves media through V2 only when the encoder token budget fits one tower call (32768 patches; the default 8192 fits). Larger budgets fall back to V1 under auto.
  • Text-only targets refuse media at admission with one message; GLM's error text changes.

Follow-ups (not in this PR).

Performance. Not measured (needs GB300). Expected: later prefill chunks of a media request no longer run the tower again, and repeated images can hit the encoder cache.

Test Coverage

  • New tests/unittest/_torch/modeling_v2/test_modeling_v2_multimodal.py: the base with a tiny language model and tower (construction: the tower built once from vision_config at the activation dtype and never with disable_mm_encoder, mm_encoder_class and the build_mm_encoder override, the copied language model flags, runtime contract, admission also through the executor, forward in both row modes through the real mixin, no tower call for a step without media or a chunk after its placeholders, the extra forward arguments hook, compile scope, weight split and weight_mapper pass-through bound to llm, the route-time check with the marker and an inherited validate_request, and the side-stream decline under auto and require, also with the encoder in the step and on an encoder-only worker).
  • test_modeling_v2_contracts.py: the multimodal contract (each text-only admission hook run on media, embeddings and M-RoPE positions; the own-encoder list) and naming rule checks, on today's targets and on broken stand-ins.
  • qwen4_exp NVFP4 test_multimodal_model.py, test_weight_shards.py, test_vision.py, test_routing.py; the FP8 target's routing test expects the NVFP4 class; GLM-5.3-Flash admission and Kimi K3 forward-check tests; has_media_or_embeddings, executor admission, registry accessor, mixin row split and breakable gate (both row modes) tests.
  • CPU, CUDA hidden, main 8ba86b8 test image running the code of main 42c1287, this PR vs main (all Modeling V2 unit tests, multimodal mixin and engine, multimodal scheduler, model engine, executor, inputs): 0 changed outcomes in 8618 common cases; 106 new tests pass; 0 changed outcomes at each of the 6 commits. The FP8 target's routing tests (native ops stubbed at import, as the image lacks them): 159 passed at every commit, as on main.
  • One GB200 GPU, all markers, this head vs main 42c1287: multimodal mixin, engine multimodal, model engine and the base tests: 237 passed (main 188, 0 changed); qwen4_exp NVFP4 multimodal, vision and routing tests: 155 passed (main 149, 0 changed, 5 moved).
  • Needs GB300 (sm_103; not run): qwen4_exp NVFP4 boot with modeling_v2="require", the breakable prefill accuracy cases with a tower, MMMU, encoder cache hits, an image across prefill chunks and MTP with images. Kimi K3, MiniMax-M3 and GLM-5.3-Flash admission is checked on CPU only.
  • Pre-commit hooks pass.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@yechank-nvidia
yechank-nvidia force-pushed the yechank/mm-modeling-v2-wrapper branch from 5b88ba4 to fa5fbca Compare November 18, 2026 01:29
@yechank-nvidia yechank-nvidia changed the title [TRTLLM-17780][feat] Run Modeling V2 vision-language models through the multimodal runtime (wrapper + text-only guards) [TRTLLM-17780][feat] Run Modeling V2 vision-language models through the multimodal runtime (shared wrapper base + text-only guards) Nov 18, 2026
@yechank-nvidia
yechank-nvidia force-pushed the yechank/mm-modeling-v2-wrapper branch from fa5fbca to 183dfcc Compare November 18, 2026 03:23
@yechank-nvidia yechank-nvidia changed the title [TRTLLM-17780][feat] Run Modeling V2 vision-language models through the multimodal runtime (shared wrapper base + text-only guards) [TRTLLM-17780][feat] Run Modeling V2 vision-language models through the multimodal runtime (V1-style composite model + text-only guards) Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, what is this for / why is it necessary?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The router and the model loader read three language model flags off the routed class, before any model is built: moe_impls_honored and draft_moe_impls_honored in the MoE tactic check (_router_index.py:247,257), and supports_nvfp4_kv_cache in the SM107 KV cache check (model_loader.py:239-240, through model_config.modeling_v2_target). For a target on this base, the routed class is this composite, not the language model. Without the copy, a set moe_config.tactic would be refused (V1 fallback under auto), and an NVFP4 KV cache on SM107 could be switched to FP8.

Today's qwen4_exp language model sets none of the three, so this is for the next composites (for example Kimi K3, whose MXFP4 targets set moe_impls_honored). I looked for something simpler: a property does not work, because both readers read the class, not an instance; asking language_model_class in each reader would touch model_loader.py and every future reader. So I kept the copy and made its comment say who reads these flags and why. It also now falls back to the readers' defaults when a subclass swaps in a language model without these flags, so a subclass never keeps its parent's. Done in 1f44eec.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self: maybe better name?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MultimodalModelMixinV2?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, for consistency, could / should we go with self.mm_encoder_class instead of a build_ pattern? Since line 110 does it for the LLM.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the two sides should look the same. The base now has mm_encoder_class, built as mm_encoder_class(vision_config, dtype), next to language_model_class. The qwen4_exp target just sets mm_encoder_class = VisionTower. build_mm_encoder stays as the override for a tower that needs more than that, for example another config or more arguments. Done in 1f44eec (and a6860ef for the qwen4_exp target).

…V2 targets

Problem
The Kimi K3, MiniMax-M3 and GLM-5.3-Flash Modeling V2 targets serve the
text half of a vision-language checkpoint. Each one treats an image
request in its own way: Kimi and MiniMax raise in forward, which fails
every request in the batch; GLM checks a fixed key list at admission,
and its FP8 target uses a different hook. The MiniMax forward check also
counts an M-RoPE-only payload as media, which a disaggregated generation
request carries.

Solution
- Add has_media_or_embeddings() in tensorrt_llm/inputs/multimodal.py:
  true when multimodal_data has any non-None entry other than
  mrope_config. Unknown keys count, so it fails closed.
- All 14 text-only targets (6 Kimi K3, 5 MiniMax-M3, 3 GLM-5.3-Flash)
  define validate_multimodal_request_data(), which the executor already
  calls at admission. It raises when the helper is true, so only that
  request fails and no budget is charged. The GLM targets move their own
  checks to the same hook and message.
- The Kimi and MiniMax forward checks stay as a backstop and use the
  same helper (Kimi SM107 through check_text_only), so a payload that
  passes admission, such as M-RoPE positions alone, also passes forward.
  The lists of stock calls (UNCERTIFIED_GENERIC_CALLS) name the helper.

Validation
- CPU, CUDA hidden, main 8ba86b8 test image running the code of main
  42c1287, 8 test targets (all Modeling V2 unit tests, multimodal mixin
  and engine, multimodal scheduler, model engine, executor, inputs),
  this commit vs main: 8636 vs 8623 cases, 0 changed outcomes; the 13
  new tests pass (the helper, executor admission, the GLM-5.3-Flash FP8
  hook). The GLM-5.3-Flash NVFP4 admission tests pass with the shared
  message, and the Kimi K3 common test checks the forward backstop on
  media, embeddings and M-RoPE positions.
- The GLM-5.3-Flash admission tests run on CPU and pass: media is
  refused and an M-RoPE-only payload passes. No GPU run: the Kimi K3 and
  MiniMax-M3 targets need sm_100, sm_103, sm_107 or 16 GPUs.
- Pre-commit hooks pass.

Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
…ng V2 target

Problem
The qwen4_exp NVFP4 Modeling V2 target ignores disable_mm_encoder. It
still declares, loads and warms up the full vision tower on every rank,
and an image request with the switch on would reach a tower the user
asked not to build.

Solution
- Move the 21 vision weight entries out of the language model table into
  VISION_WEIGHTS, a second table with the same declare / manifest / read
  passes and its own load. The load checks coverage both ways: every
  tower parameter is written and every model.visual key is read. The
  language model table now always leaves model.visual keys unread.
- VisionTower is an nn.Module that declares and loads its own table and
  builds its views and warms up in post_load_weights. The shell builds
  it only when disable_mm_encoder is off.
- Without a tower, validate_request skips the geometry checks and
  validate_multimodal_request_data refuses media or embeddings, so only
  that request fails. M-RoPE-only payloads still pass.

Validation
- CPU, CUDA hidden, main 8ba86b8 test image running the code of main
  42c1287, 8 test targets (all Modeling V2 unit tests, multimodal mixin
  and engine, multimodal scheduler, model engine, executor, inputs),
  this commit vs the one before: 8641 vs 8636 cases, 0 changed outcomes;
  the 5 new tests pass (table split, vision keys left by the language
  table, tower load with two-way coverage, admission without a tower).
- The checkpoint-backed coverage tests are updated for the second table;
  they skip without the checkpoint, as before.
- Pre-commit hooks pass. qwen4_exp GPU runs need GB300 (sm_103); not
  run.

Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
…ing fusion

Problem
MultimodalModelMixin.prepare_multimodal_inputs always fuses the encoder
rows into text embeddings. A language model that places the rows itself,
at the step's mm_token_indices, and must keep input_ids (for example to
hash token ids) cannot reuse the mixin's selection, encoder cache and
chunk slicing. The Modeling V2 README also does not say how a
vision-language target may use the shared multimodal mixins.

Solution
- Add prepare_multimodal_rows(): the existing steps of
  prepare_multimodal_inputs up to the fuse call (select params, cache or
  encode, the two embedding hooks, this chunk's rows). It returns
  (input_ids, active_embeddings, extra_embeds), or None for a text step.
- prepare_multimodal_inputs() calls it, then fuses as before. V1 models
  keep the same calls in the same order.
- Modeling V2 README: MultimodalModelMixin and MultimodalEncoderMixin
  are runtime interfaces a vision-language target may inherit; it still
  uses no Modeling V1 model code. Model lifecycle names the two target
  shapes.

Validation
- CPU, CUDA hidden, main 8ba86b8 test image running the code of main
  42c1287, 8 test targets (all Modeling V2 unit tests, multimodal mixin
  and engine, multimodal scheduler, model engine, executor, inputs),
  this commit vs the one before: 8643 vs 8641 cases, 0 changed outcomes;
  the 2 new mixin tests pass (fusing the rows equals
  prepare_multimodal_inputs; a text step has no rows).
- Pre-commit hooks pass.

Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
…odeling V2 vision-language models

Problem
A Modeling V2 vision-language target needs the composite shape a V1
vision-language model has: the language model as llm and the tower as
mm_encoder behind MultimodalModelMixin. Written by hand, every target
repeats the code that hands llm what the engine reads, and a new target
can miss a piece. Nothing checks that a class routed for a
vision-language architecture serves media or refuses it.

Solution
- Add ModelingV2ForConditionalGeneration in modeling_v2/multimodal.py:
  MultimodalModelMixin plus PreTrainedModel, the same composite shape as
  a Modeling V1 vision-language model (for example Qwen3VLModelBase). It
  holds only the code every V2 target needs. A target subclasses it as
  ModelingV2<Target>ForConditionalGeneration and declares its language
  model class, its tower class, the tower's weight prefix, its encoder
  hook and how rows reach the language model. The base does the rest
  once:
  - builds llm on a config copy (skip_create_weights_in_init) and keeps
    the engine's ModelConfig, which ends with the text config;
  - builds the tower as mm_encoder_class(vision_config, dtype), as it
    builds llm from language_model_class; a tower built another way
    overrides build_mm_encoder. No tower with disable_mm_encoder;
  - forwards lm_head, draft_config, draft_model, spec_worker,
    multimodal_token_ids and the KV cache manager hook, and copies the
    language model flags that the router (MoE tactics) and the loader
    (NVFP4 KV cache) read off the routed class, or the readers' defaults
    when the language model has none, so a subclass never keeps its
    parent's;
  - compiles only llm.model;
  - at admission: checks the token range (the executor does this for
    decoder models only) and an optional tower hook, refuses precomputed
    embeddings, and refuses media without a tower;
  - selects only requests with media, so M-RoPE-only params never reach
    the tower;
  - gives the tower the keys under its prefix and llm the rest, and
    leaves those keys unread without a tower. llm gets the engine's
    weight_mapper when its load_weights takes one (call_load_weights),
    bound to llm first, as the V1 composite models do, since the keys
    name llm's modules.
  The base imports no routing module.
- forward passes the rows fused into inputs_embeds (default), or, with
  language_model_takes_rows, as multimodal_rows=(rows, mm_token_indices)
  beside unchanged input_ids. In both modes it calls the mixin's
  get_language_model_extra_forward_kwargs hook as the mixin's own
  forward does and passes what it returns to llm (V1 Qwen3-VL passes its
  DeepStack rows this way). The fused mode takes the mixin's inputs as
  they are, also with caller inputs_embeds; the rows mode keeps the
  extra rows also when the chunk has no multimodal row. The default hook
  returns nothing.
- The router checks every class a route returns. When the architecture's
  input processor takes media, the class must be a MultimodalModelMixin
  model (the base is one) or set the supports_multimodal marker, define
  validate_multimodal_request_data (text only), or define
  validate_request in its own class (a target that encodes media in its
  own forward; an inherited validate_request does not count).
  Otherwise startup fails with a ValueError, under auto too. The router
  reads the input processor through a new public accessor,
  InputProcessorRegistry.get_input_processor_cls, which
  create_input_processor now uses too.
- The router declines a base subclass that builds its tower while
  encoder_side_stream_max_ahead > 0 (NoTarget, so auto uses the built-in
  model). An encoder-only worker (mm_encoder_only) is not declined: it
  builds the architecture's encoder model, not this one.
- The breakable prefill gate also accepts a composite whose llm is a
  decoder and that sets language_model_takes_rows: llm takes input_ids
  and places the rows itself, as a decoder routed alone does. A
  composite that fuses rows into inputs_embeds is still refused.

Validation
- CPU, CUDA hidden, main 8ba86b8 test image running the code of main
  42c1287, 8 test targets (all Modeling V2 unit tests, multimodal mixin
  and engine, multimodal scheduler, model engine, executor, inputs),
  this commit vs the one before: 8691 vs 8643 cases, 0 changed outcomes;
  the 48 new tests pass: the base (construction, with the tower built
  once from vision_config at the activation dtype and never with
  disable_mm_encoder, mm_encoder_class and the build_mm_encoder
  override, the copied language model flags, runtime contract, admission
  also through the executor, forward in both row modes through the real
  mixin, no tower call for a step without media or a chunk after its
  placeholders, the extra forward arguments hook, compile scope, weight
  split, and weight_mapper pass-through bound to llm), the router check
  (marker True and False, an inherited validate_request), the
  side-stream decline under auto and require (also with the encoder in
  the step and on an encoder-only worker), the breakable gate in both
  row modes and the registry accessor.
- One GB200 GPU, all markers, on the series head (the later commits
  change none of these files): multimodal mixin, engine multimodal,
  model engine and the base tests: 237 passed, main 42c1287 188, 0
  changed outcomes.
- Pre-commit hooks pass.

Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
…ditionalGeneration base

Problem
The qwen4_exp NVFP4 Modeling V2 target is a DecoderModelForCausalLM that
runs its vision tower inside the language model forward. The multimodal
runtime does not see it as a multimodal model: no encoder cache, no
encoder memory reserve at startup, and the tower runs again on the whole
request for every prefill chunk with media.

Solution
- Route Qwen4ExpForConditionalGeneration to a 29-line subclass of
  ModelingV2ForConditionalGeneration,
  ModelingV2Qwen38FlashNextNvfp4Sm103DefaultForConditionalGeneration. It
  declares the language model target, the tower class (VisionTower,
  which the base builds), the model.visual. weight prefix, the encoder
  hook and the encoder cache. It takes the rows mode: the PLE layer
  hashes input_ids, so the language model writes the rows at
  mm_token_indices in its own prefill step.
- VisionTower is also a MultimodalEncoderMixin. It builds its own
  attention arguments per call, so setup_attn_metadata is a no-op.
- check_deployment declines an encoder token budget over 32768 while the
  tower is built: startup profiling fills the budget with dummy images,
  and one tower call takes at most 32768 patches. auto then uses
  Modeling V1; require fails and names the setting. The default budget
  (8192) fits.
- vision.encode gives one request's rows, whole and in prompt order. It
  needs no M-RoPE ids, so profiling can encode its dummy image. The
  mixin keeps the chunk's rows; vision.multimodal_embeds is removed.
- The language model takes multimodal_rows and no longer builds, loads
  or checks the tower. The tower's admission check runs from
  validate_media_request.
- Breakable prefill runs as on main, with or without a tower: the
  subclass takes the rows mode, which the engine's gate accepts. The
  breakable accuracy cases do not change.
- The FP8 target's routing test now expects this class for an NVFP4
  checkpoint.
- The FP8 target does not change: it encodes only the frames of the
  current prefill chunk in its own forward.

Validation
- CPU, CUDA hidden, main 8ba86b8 test image running the code of main
  42c1287, 8 test targets (all Modeling V2 unit tests, multimodal mixin
  and engine, multimodal scheduler, model engine, executor, inputs),
  this commit vs the one before: 8696 vs 8691 cases, 0 changed outcomes;
  11 new tests pass (what the subclass declares, the real tower's
  construction and weight keys, the budget decline under auto and
  require, the tower's admission check, the encoder hook, the rows
  handed to the language model); 6 tests of the removed shell
  bookkeeping moved there or to the base tests.
- The FP8 target's routing tests on CPU, with only its native op library
  stubbed at import (this image does not ship it): 159 passed, as on
  main.
- One GB200 GPU, all markers, on the series head (the last commit
  changes no qwen4_exp file): qwen4_exp NVFP4 test_multimodal_model,
  test_vision, test_vision_warm_up and test_routing: 155 passed; main
  42c1287 149 passed, 0 changed outcomes, 5 moved.
- Pre-commit hooks pass.
- Not run: every qwen4_exp GPU path needs GB300 (sm_103): boot at each
  layout, MMMU, encoder cache hits, chunked images, MTP with images and
  the breakable prefill accuracy cases with a tower.

Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
…ntract

Problem
The router checks a routed class at startup, which needs the target's
GPU. Nothing finds a broken vision-language target on a CPU machine, and
the README does not say how to add one or how to name it.

Solution
- test_modeling_v2_contracts.py reads every class a route returns and
  checks those routed for an architecture whose input processor takes
  media (the router's own test, not the class name). A route return that
  is not a class imported from a target's modeling module fails the
  check, so no routed class is skipped. Each checked class is either a
  ModelingV2ForConditionalGeneration subclass that sets
  language_model_class, mm_encoder_weight_prefix and its tower
  (mm_encoder_class set to a class, or a build_mm_encoder override) and
  defines encode_multimodal_inputs, or a text-only target
  whose validate_multimodal_request_data is run on its own with the real
  has_media_or_embeddings: it must refuse media, embeddings and unknown
  keys with the shared message and admit M-RoPE positions alone. A
  target that encodes media in its own forward is listed by name (today
  only the qwen4_exp FP8 target) and must define a validate_request that
  refuses something. An entry fails when it is no longer routed for
  media, or when its class moved onto the base or serves text only, so
  the list only shrinks. A forward that refuses media uses the same
  helper. Fixture families show each broken case is reported.
- The same file checks the naming rule of V2 vision-language models:
  every routed class that is a multimodal model (by its base) subclasses
  ModelingV2ForConditionalGeneration and its name ends with
  ForConditionalGeneration, and no class in a target's modeling.py is
  named ...VLM. The ModelingV2<Target> part is left to review. A
  stand-in that breaks each part is reported.
- Modeling V2 README: a "Multimodal targets" section with the base, a
  short template, the two row modes, the hook for extra language model
  inputs, the text-only rule, the listed targets that encode media
  themselves, disable_mm_encoder, breakable prefill and the encoder side
  stream with a tower, and the rule: code every V2 vision-language model
  needs lives in the one shared base, and a class that serves media is
  ModelingV2<Target>ForConditionalGeneration (never ...VLM or ...Model)
  while its language model target stays ModelingV2<Target>. It says
  which parts the test checks. The code layout, the Modeling V1 code
  rule, the naming section and the lifecycle name the base.
  adding_new_model.md states the rule in a short "Multimodal Models"
  part and points to the section.

Validation
- CPU, CUDA hidden, main 8ba86b8 test image running the code of main
  42c1287, 8 test targets (all Modeling V2 unit tests, multimodal mixin
  and engine, multimodal scheduler, model engine, executor, inputs),
  this commit vs the one before: 8724 vs 8696 cases, 0 changed outcomes;
  the 28 new contract tests pass on today's targets and report each
  broken stand-in.
- This series vs main 42c1287: 8618 common cases, 0 changed outcomes,
  106 new tests pass, 5 moved.
- Pre-commit hooks pass.

Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we made _vision.encode a method of mm_encoder (e.g. forward), couldn't this entire method be in the ModelingV2ForConditionalGeneration base class?

}


class ModelingV2ForConditionalGeneration(MultimodalModelMixin, PreTrainedModel):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Investigate whether we can remove PreTrainedModel altogether.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants