Repository navigation
[TRTLLM-17780][feat] Run Modeling V2 vision-language models through the multimodal runtime (V1-style composite model + text-only guards) - #22450
Conversation
5b88ba4 to
fa5fbca
Compare
fa5fbca to
183dfcc
Compare
There was a problem hiding this comment.
Out of curiosity, what is this for / why is it necessary?
There was a problem hiding this comment.
The router and the model loader read three language model flags off the routed class, before any model is built: moe_impls_honored and draft_moe_impls_honored in the MoE tactic check (_router_index.py:247,257), and supports_nvfp4_kv_cache in the SM107 KV cache check (model_loader.py:239-240, through model_config.modeling_v2_target). For a target on this base, the routed class is this composite, not the language model. Without the copy, a set moe_config.tactic would be refused (V1 fallback under auto), and an NVFP4 KV cache on SM107 could be switched to FP8.
Today's qwen4_exp language model sets none of the three, so this is for the next composites (for example Kimi K3, whose MXFP4 targets set moe_impls_honored). I looked for something simpler: a property does not work, because both readers read the class, not an instance; asking language_model_class in each reader would touch model_loader.py and every future reader. So I kept the copy and made its comment say who reads these flags and why. It also now falls back to the readers' defaults when a subclass swaps in a language model without these flags, so a subclass never keeps its parent's. Done in 1f44eec.
There was a problem hiding this comment.
Note to self: maybe better name?
There was a problem hiding this comment.
Just curious, for consistency, could / should we go with self.mm_encoder_class instead of a build_ pattern? Since line 110 does it for the LLM.
There was a problem hiding this comment.
Agreed, the two sides should look the same. The base now has mm_encoder_class, built as mm_encoder_class(vision_config, dtype), next to language_model_class. The qwen4_exp target just sets mm_encoder_class = VisionTower. build_mm_encoder stays as the override for a tower that needs more than that, for example another config or more arguments. Done in 1f44eec (and a6860ef for the qwen4_exp target).
…V2 targets Problem The Kimi K3, MiniMax-M3 and GLM-5.3-Flash Modeling V2 targets serve the text half of a vision-language checkpoint. Each one treats an image request in its own way: Kimi and MiniMax raise in forward, which fails every request in the batch; GLM checks a fixed key list at admission, and its FP8 target uses a different hook. The MiniMax forward check also counts an M-RoPE-only payload as media, which a disaggregated generation request carries. Solution - Add has_media_or_embeddings() in tensorrt_llm/inputs/multimodal.py: true when multimodal_data has any non-None entry other than mrope_config. Unknown keys count, so it fails closed. - All 14 text-only targets (6 Kimi K3, 5 MiniMax-M3, 3 GLM-5.3-Flash) define validate_multimodal_request_data(), which the executor already calls at admission. It raises when the helper is true, so only that request fails and no budget is charged. The GLM targets move their own checks to the same hook and message. - The Kimi and MiniMax forward checks stay as a backstop and use the same helper (Kimi SM107 through check_text_only), so a payload that passes admission, such as M-RoPE positions alone, also passes forward. The lists of stock calls (UNCERTIFIED_GENERIC_CALLS) name the helper. Validation - CPU, CUDA hidden, main 8ba86b8 test image running the code of main 42c1287, 8 test targets (all Modeling V2 unit tests, multimodal mixin and engine, multimodal scheduler, model engine, executor, inputs), this commit vs main: 8636 vs 8623 cases, 0 changed outcomes; the 13 new tests pass (the helper, executor admission, the GLM-5.3-Flash FP8 hook). The GLM-5.3-Flash NVFP4 admission tests pass with the shared message, and the Kimi K3 common test checks the forward backstop on media, embeddings and M-RoPE positions. - The GLM-5.3-Flash admission tests run on CPU and pass: media is refused and an M-RoPE-only payload passes. No GPU run: the Kimi K3 and MiniMax-M3 targets need sm_100, sm_103, sm_107 or 16 GPUs. - Pre-commit hooks pass. Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
…ng V2 target Problem The qwen4_exp NVFP4 Modeling V2 target ignores disable_mm_encoder. It still declares, loads and warms up the full vision tower on every rank, and an image request with the switch on would reach a tower the user asked not to build. Solution - Move the 21 vision weight entries out of the language model table into VISION_WEIGHTS, a second table with the same declare / manifest / read passes and its own load. The load checks coverage both ways: every tower parameter is written and every model.visual key is read. The language model table now always leaves model.visual keys unread. - VisionTower is an nn.Module that declares and loads its own table and builds its views and warms up in post_load_weights. The shell builds it only when disable_mm_encoder is off. - Without a tower, validate_request skips the geometry checks and validate_multimodal_request_data refuses media or embeddings, so only that request fails. M-RoPE-only payloads still pass. Validation - CPU, CUDA hidden, main 8ba86b8 test image running the code of main 42c1287, 8 test targets (all Modeling V2 unit tests, multimodal mixin and engine, multimodal scheduler, model engine, executor, inputs), this commit vs the one before: 8641 vs 8636 cases, 0 changed outcomes; the 5 new tests pass (table split, vision keys left by the language table, tower load with two-way coverage, admission without a tower). - The checkpoint-backed coverage tests are updated for the second table; they skip without the checkpoint, as before. - Pre-commit hooks pass. qwen4_exp GPU runs need GB300 (sm_103); not run. Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
…ing fusion Problem MultimodalModelMixin.prepare_multimodal_inputs always fuses the encoder rows into text embeddings. A language model that places the rows itself, at the step's mm_token_indices, and must keep input_ids (for example to hash token ids) cannot reuse the mixin's selection, encoder cache and chunk slicing. The Modeling V2 README also does not say how a vision-language target may use the shared multimodal mixins. Solution - Add prepare_multimodal_rows(): the existing steps of prepare_multimodal_inputs up to the fuse call (select params, cache or encode, the two embedding hooks, this chunk's rows). It returns (input_ids, active_embeddings, extra_embeds), or None for a text step. - prepare_multimodal_inputs() calls it, then fuses as before. V1 models keep the same calls in the same order. - Modeling V2 README: MultimodalModelMixin and MultimodalEncoderMixin are runtime interfaces a vision-language target may inherit; it still uses no Modeling V1 model code. Model lifecycle names the two target shapes. Validation - CPU, CUDA hidden, main 8ba86b8 test image running the code of main 42c1287, 8 test targets (all Modeling V2 unit tests, multimodal mixin and engine, multimodal scheduler, model engine, executor, inputs), this commit vs the one before: 8643 vs 8641 cases, 0 changed outcomes; the 2 new mixin tests pass (fusing the rows equals prepare_multimodal_inputs; a text step has no rows). - Pre-commit hooks pass. Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
…odeling V2 vision-language models
Problem
A Modeling V2 vision-language target needs the composite shape a V1
vision-language model has: the language model as llm and the tower as
mm_encoder behind MultimodalModelMixin. Written by hand, every target
repeats the code that hands llm what the engine reads, and a new target
can miss a piece. Nothing checks that a class routed for a
vision-language architecture serves media or refuses it.
Solution
- Add ModelingV2ForConditionalGeneration in modeling_v2/multimodal.py:
MultimodalModelMixin plus PreTrainedModel, the same composite shape as
a Modeling V1 vision-language model (for example Qwen3VLModelBase). It
holds only the code every V2 target needs. A target subclasses it as
ModelingV2<Target>ForConditionalGeneration and declares its language
model class, its tower class, the tower's weight prefix, its encoder
hook and how rows reach the language model. The base does the rest
once:
- builds llm on a config copy (skip_create_weights_in_init) and keeps
the engine's ModelConfig, which ends with the text config;
- builds the tower as mm_encoder_class(vision_config, dtype), as it
builds llm from language_model_class; a tower built another way
overrides build_mm_encoder. No tower with disable_mm_encoder;
- forwards lm_head, draft_config, draft_model, spec_worker,
multimodal_token_ids and the KV cache manager hook, and copies the
language model flags that the router (MoE tactics) and the loader
(NVFP4 KV cache) read off the routed class, or the readers' defaults
when the language model has none, so a subclass never keeps its
parent's;
- compiles only llm.model;
- at admission: checks the token range (the executor does this for
decoder models only) and an optional tower hook, refuses precomputed
embeddings, and refuses media without a tower;
- selects only requests with media, so M-RoPE-only params never reach
the tower;
- gives the tower the keys under its prefix and llm the rest, and
leaves those keys unread without a tower. llm gets the engine's
weight_mapper when its load_weights takes one (call_load_weights),
bound to llm first, as the V1 composite models do, since the keys
name llm's modules.
The base imports no routing module.
- forward passes the rows fused into inputs_embeds (default), or, with
language_model_takes_rows, as multimodal_rows=(rows, mm_token_indices)
beside unchanged input_ids. In both modes it calls the mixin's
get_language_model_extra_forward_kwargs hook as the mixin's own
forward does and passes what it returns to llm (V1 Qwen3-VL passes its
DeepStack rows this way). The fused mode takes the mixin's inputs as
they are, also with caller inputs_embeds; the rows mode keeps the
extra rows also when the chunk has no multimodal row. The default hook
returns nothing.
- The router checks every class a route returns. When the architecture's
input processor takes media, the class must be a MultimodalModelMixin
model (the base is one) or set the supports_multimodal marker, define
validate_multimodal_request_data (text only), or define
validate_request in its own class (a target that encodes media in its
own forward; an inherited validate_request does not count).
Otherwise startup fails with a ValueError, under auto too. The router
reads the input processor through a new public accessor,
InputProcessorRegistry.get_input_processor_cls, which
create_input_processor now uses too.
- The router declines a base subclass that builds its tower while
encoder_side_stream_max_ahead > 0 (NoTarget, so auto uses the built-in
model). An encoder-only worker (mm_encoder_only) is not declined: it
builds the architecture's encoder model, not this one.
- The breakable prefill gate also accepts a composite whose llm is a
decoder and that sets language_model_takes_rows: llm takes input_ids
and places the rows itself, as a decoder routed alone does. A
composite that fuses rows into inputs_embeds is still refused.
Validation
- CPU, CUDA hidden, main 8ba86b8 test image running the code of main
42c1287, 8 test targets (all Modeling V2 unit tests, multimodal mixin
and engine, multimodal scheduler, model engine, executor, inputs),
this commit vs the one before: 8691 vs 8643 cases, 0 changed outcomes;
the 48 new tests pass: the base (construction, with the tower built
once from vision_config at the activation dtype and never with
disable_mm_encoder, mm_encoder_class and the build_mm_encoder
override, the copied language model flags, runtime contract, admission
also through the executor, forward in both row modes through the real
mixin, no tower call for a step without media or a chunk after its
placeholders, the extra forward arguments hook, compile scope, weight
split, and weight_mapper pass-through bound to llm), the router check
(marker True and False, an inherited validate_request), the
side-stream decline under auto and require (also with the encoder in
the step and on an encoder-only worker), the breakable gate in both
row modes and the registry accessor.
- One GB200 GPU, all markers, on the series head (the later commits
change none of these files): multimodal mixin, engine multimodal,
model engine and the base tests: 237 passed, main 42c1287 188, 0
changed outcomes.
- Pre-commit hooks pass.
Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
…ditionalGeneration base Problem The qwen4_exp NVFP4 Modeling V2 target is a DecoderModelForCausalLM that runs its vision tower inside the language model forward. The multimodal runtime does not see it as a multimodal model: no encoder cache, no encoder memory reserve at startup, and the tower runs again on the whole request for every prefill chunk with media. Solution - Route Qwen4ExpForConditionalGeneration to a 29-line subclass of ModelingV2ForConditionalGeneration, ModelingV2Qwen38FlashNextNvfp4Sm103DefaultForConditionalGeneration. It declares the language model target, the tower class (VisionTower, which the base builds), the model.visual. weight prefix, the encoder hook and the encoder cache. It takes the rows mode: the PLE layer hashes input_ids, so the language model writes the rows at mm_token_indices in its own prefill step. - VisionTower is also a MultimodalEncoderMixin. It builds its own attention arguments per call, so setup_attn_metadata is a no-op. - check_deployment declines an encoder token budget over 32768 while the tower is built: startup profiling fills the budget with dummy images, and one tower call takes at most 32768 patches. auto then uses Modeling V1; require fails and names the setting. The default budget (8192) fits. - vision.encode gives one request's rows, whole and in prompt order. It needs no M-RoPE ids, so profiling can encode its dummy image. The mixin keeps the chunk's rows; vision.multimodal_embeds is removed. - The language model takes multimodal_rows and no longer builds, loads or checks the tower. The tower's admission check runs from validate_media_request. - Breakable prefill runs as on main, with or without a tower: the subclass takes the rows mode, which the engine's gate accepts. The breakable accuracy cases do not change. - The FP8 target's routing test now expects this class for an NVFP4 checkpoint. - The FP8 target does not change: it encodes only the frames of the current prefill chunk in its own forward. Validation - CPU, CUDA hidden, main 8ba86b8 test image running the code of main 42c1287, 8 test targets (all Modeling V2 unit tests, multimodal mixin and engine, multimodal scheduler, model engine, executor, inputs), this commit vs the one before: 8696 vs 8691 cases, 0 changed outcomes; 11 new tests pass (what the subclass declares, the real tower's construction and weight keys, the budget decline under auto and require, the tower's admission check, the encoder hook, the rows handed to the language model); 6 tests of the removed shell bookkeeping moved there or to the base tests. - The FP8 target's routing tests on CPU, with only its native op library stubbed at import (this image does not ship it): 159 passed, as on main. - One GB200 GPU, all markers, on the series head (the last commit changes no qwen4_exp file): qwen4_exp NVFP4 test_multimodal_model, test_vision, test_vision_warm_up and test_routing: 155 passed; main 42c1287 149 passed, 0 changed outcomes, 5 moved. - Pre-commit hooks pass. - Not run: every qwen4_exp GPU path needs GB300 (sm_103): boot at each layout, MMMU, encoder cache hits, chunked images, MTP with images and the breakable prefill accuracy cases with a tower. Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
…ntract Problem The router checks a routed class at startup, which needs the target's GPU. Nothing finds a broken vision-language target on a CPU machine, and the README does not say how to add one or how to name it. Solution - test_modeling_v2_contracts.py reads every class a route returns and checks those routed for an architecture whose input processor takes media (the router's own test, not the class name). A route return that is not a class imported from a target's modeling module fails the check, so no routed class is skipped. Each checked class is either a ModelingV2ForConditionalGeneration subclass that sets language_model_class, mm_encoder_weight_prefix and its tower (mm_encoder_class set to a class, or a build_mm_encoder override) and defines encode_multimodal_inputs, or a text-only target whose validate_multimodal_request_data is run on its own with the real has_media_or_embeddings: it must refuse media, embeddings and unknown keys with the shared message and admit M-RoPE positions alone. A target that encodes media in its own forward is listed by name (today only the qwen4_exp FP8 target) and must define a validate_request that refuses something. An entry fails when it is no longer routed for media, or when its class moved onto the base or serves text only, so the list only shrinks. A forward that refuses media uses the same helper. Fixture families show each broken case is reported. - The same file checks the naming rule of V2 vision-language models: every routed class that is a multimodal model (by its base) subclasses ModelingV2ForConditionalGeneration and its name ends with ForConditionalGeneration, and no class in a target's modeling.py is named ...VLM. The ModelingV2<Target> part is left to review. A stand-in that breaks each part is reported. - Modeling V2 README: a "Multimodal targets" section with the base, a short template, the two row modes, the hook for extra language model inputs, the text-only rule, the listed targets that encode media themselves, disable_mm_encoder, breakable prefill and the encoder side stream with a tower, and the rule: code every V2 vision-language model needs lives in the one shared base, and a class that serves media is ModelingV2<Target>ForConditionalGeneration (never ...VLM or ...Model) while its language model target stays ModelingV2<Target>. It says which parts the test checks. The code layout, the Modeling V1 code rule, the naming section and the lifecycle name the base. adding_new_model.md states the rule in a short "Multimodal Models" part and points to the section. Validation - CPU, CUDA hidden, main 8ba86b8 test image running the code of main 42c1287, 8 test targets (all Modeling V2 unit tests, multimodal mixin and engine, multimodal scheduler, model engine, executor, inputs), this commit vs the one before: 8724 vs 8696 cases, 0 changed outcomes; the 28 new contract tests pass on today's targets and report each broken stand-in. - This series vs main 42c1287: 8618 common cases, 0 changed outcomes, 106 new tests pass, 5 moved. - Pre-commit hooks pass. Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
183dfcc to
e273bcb
Compare
There was a problem hiding this comment.
If we made _vision.encode a method of mm_encoder (e.g. forward), couldn't this entire method be in the ModelingV2ForConditionalGeneration base class?
| } | ||
|
|
||
|
|
||
| class ModelingV2ForConditionalGeneration(MultimodalModelMixin, PreTrainedModel): |
There was a problem hiding this comment.
Investigate whether we can remove PreTrainedModel altogether.
@coderabbitai summary
Description
Why. Modeling V2 routes four vision-language architectures, but none of their targets runs as a multimodal model. The qwen4_exp NVFP4 target runs its vision tower inside the language model forward, so it gets no encoder cache and no encoder memory reserve at startup, and it runs the tower again on the whole request for every prefill chunk that has media. The Kimi K3, MiniMax-M3 and GLM-5.3-Flash targets serve text only, and each one treats an image request in its own way: some fail the whole batch in forward, some check a fixed key list. This PR gives a V2 vision-language target the same composite shape V1 vision-language models have, through one shared base class, and makes every text-only target refuse media at admission with one shared helper. No new flag, env var or config.
Based on
main. The new base inheritsMultimodalModelMixin, so every executor check onmain(isinstance(model, MultimodalModelMixin)) already accepts it. The route-time check also accepts a class with thesupports_multimodalmarker of #20906.Class shape.
What.
validate_multimodal_request_dataraises when the new helperhas_media_or_embeddings()is true. Only that request fails. M-RoPE-only payloads (disaggregated generation) pass. The GLM targets move their own checks to this hook and helper.disable_mm_encoder: no tower, no vision weights. The vision weights move to their own table, loaded by the tower.prepare_multimodal_rows()is split out ofprepare_multimodal_inputs(), which now calls it and then fuses. V1 models keep the same calls in the same order.ModelingV2ForConditionalGeneration. It buildsllmon a config copy fromlanguage_model_classand the tower frommm_encoder_class(asmm_encoder_class(vision_config, dtype); a tower built another way overridesbuild_mm_encoder), forwards what the engine reads (lm_head,draft_*,spec_worker,multimodal_token_ids, class hooks), compiles onlyllm.model, checks the token range at admission, selects only requests with media, and splits the weights by the tower's key prefix.llmgets the engine'sweight_mapper, bound tollm, when itsload_weightstakes one. The rows reachllmfused intoinputs_embeds(default) or, withlanguage_model_takes_rows, as(rows, mm_token_indices)beside unchangedinput_ids. The base imports no routing module.flowchart LR P["Step"] --> S{"Mixin: requests<br/>with media?"} S -->|"none"| LLM["llm.forward"] S -->|"some"| C{"Encoder cache /<br/>earlier chunk"} C -->|hit| ROWS["Encoder rows"] C -->|miss| ENC["mm_encoder"] --> ROWS ROWS -->|"default"| F["Fuse into inputs_embeds"] --> LLM ROWS -->|"language_model_takes_rows"| K["input_ids kept +<br/>(rows, mm_token_indices)"] --> LLM X["get_language_model_extra_forward_kwargs<br/>(e.g. DeepStack, M-RoPE)"] -.-> LLMMultimodalModelMixinmodel or thesupports_multimodalmarker), have the text-only admission hook (validate_multimodal_request_data), or encode media in its own forward and have avalidate_requestof its own class (an inherited one does not count; today only the qwen4_exp FP8 target, an own-encoder target that later becomes aselected_rowsadapter; this PR does not change it). Otherwise startup fails with aValueError, underautotoo. Startup checks that the hook is there; the contract test below runs each text-only hook on media. The router finds the input processor through a new public accessor,INPUT_PROCESSOR_REGISTRY.get_input_processor_cls().encoder_side_stream_max_ahead > 0, soautouses the built-in model. An encoder-only worker (mm_encoder_only) is not declined: it builds the architecture's encoder model, not this one.llmis a decoder and that setslanguage_model_takes_rows, so qwen4_exp NVFP4 with a tower keeps the breakable prefill graph, as onmain.Rule for V2 vision-language models. A V2 vision-language model has the composite shape of a V1 one: the V2 language model as
llmand the tower asmm_encoder, run byMultimodalModelMixin. Code that every V2 vision-language model needs lives in one shared base,ModelingV2ForConditionalGeneration; a target declares only what differs, and code that several targets would copy goes into the base. A V2 class that serves media through the shared runtime is namedModelingV2<Target>ForConditionalGenerationand subclasses the base, never...VLMor...Model. The qwen4_exp FP8 target, which encodes media in its own forward, is the one listed exception. Language model targets keep theirModelingV2<Target>names. Kimi K3, GLM-5.3-Flash and MiniMax-M3 serve text only and keep their names; when they get media, they follow this rule.test_modeling_v2_contracts.pychecks that such a class subclasses the base and that its name ends withForConditionalGeneration, and that no target class ends withVLM; review checks theModelingV2<Target>part.Since the last push. Rebased onto
main(42c1287) without #20906 (34 own files before, 45 now). Main added five text-only targets (4 Kimi K3 NVFP4, 1 GLM-5.3-Flash FP8) and the qwen4_exp FP8 target; the text-only ones now use the shared admission hook, and the FP8 target is accepted as a target that encodes media itself. The side-stream decline moved from the base constructor to the router, and the base now passesweight_mapperon. qwen4_exp NVFP4 keeps breakable prefill with a tower, as onmain. From review: the base takes the tower asmm_encoder_class, built asmm_encoder_class(vision_config, dtype)next tolanguage_model_class, andbuild_mm_encoderstays as the override for a tower built another way; the comment on the copied language model flags now says who reads them, and a subclass that swaps in another language model takes that model's flags (or the readers' defaults), never its parent's.Behavior changes.
encoder_side_stream_max_ahead > 0with a built tower now falls back to V1 underauto(fails underrequire).multimodal_embeddinginput at admission (it was ignored before).auto.Follow-ups (not in this PR).
inputs_embeds, after [None][perf] Enable breakable prefill CUDA graphs for multimodal models, for text and media requests #21333.known_gaps.pyentries the first makes stale (12 text-only admission keys, the qwen4_exp NVFP4 base gap and 3 of its feature gaps) and lowers the sizes intest_contract.py.Performance. Not measured (needs GB300). Expected: later prefill chunks of a media request no longer run the tower again, and repeated images can hit the encoder cache.
Test Coverage
tests/unittest/_torch/modeling_v2/test_modeling_v2_multimodal.py: the base with a tiny language model and tower (construction: the tower built once fromvision_configat the activation dtype and never withdisable_mm_encoder,mm_encoder_classand thebuild_mm_encoderoverride, the copied language model flags, runtime contract, admission also through the executor, forward in both row modes through the real mixin, no tower call for a step without media or a chunk after its placeholders, the extra forward arguments hook, compile scope, weight split andweight_mapperpass-through bound tollm, the route-time check with the marker and an inheritedvalidate_request, and the side-stream decline underautoandrequire, also with the encoder in the step and on an encoder-only worker).test_modeling_v2_contracts.py: the multimodal contract (each text-only admission hook run on media, embeddings and M-RoPE positions; the own-encoder list) and naming rule checks, on today's targets and on broken stand-ins.test_multimodal_model.py,test_weight_shards.py,test_vision.py,test_routing.py; the FP8 target's routing test expects the NVFP4 class; GLM-5.3-Flash admission and Kimi K3 forward-check tests;has_media_or_embeddings, executor admission, registry accessor, mixin row split and breakable gate (both row modes) tests.main8ba86b8 test image running the code ofmain42c1287, this PR vsmain(all Modeling V2 unit tests, multimodal mixin and engine, multimodal scheduler, model engine, executor, inputs): 0 changed outcomes in 8618 common cases; 106 new tests pass; 0 changed outcomes at each of the 6 commits. The FP8 target's routing tests (native ops stubbed at import, as the image lacks them): 159 passed at every commit, as onmain.main42c1287: multimodal mixin, engine multimodal, model engine and the base tests: 237 passed (main 188, 0 changed); qwen4_exp NVFP4 multimodal, vision and routing tests: 155 passed (main 149, 0 changed, 5 moved).modeling_v2="require", the breakable prefill accuracy cases with a tower, MMMU, encoder cache hits, an image across prefill chunks and MTP with images. Kimi K3, MiniMax-M3 and GLM-5.3-Flash admission is checked on CPU only.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.