Repository navigation
[None][feat] Add a multimodal model protocol and CPU checks for required runtime features - #20906
Draft
yechank-nvidia wants to merge 3 commits into
Draft
yechank-nvidia wants to merge 3 commits into
yechank-nvidia wants to merge 3 commits into
Conversation
1 task done
yechank-nvidia
force-pushed
the
refactor/mm-protocol-20261023
branch
from
November 12, 2026 06:31
5c0e8c7 to
253e646
Compare
Collaborator
|
Hi, heads-up: #22286 has merged, so ModelingV2 now lives at Since this PR touches modeling_v2, please rebase it onto the latest main. You can use these migration commands to move any files you added under the old directory and update any remaining references to the old path. |
yechank-nvidia
force-pushed
the
refactor/mm-protocol-20261023
branch
from
November 17, 2026 08:22
253e646 to
d8f213e
Compare
1 task done
Problem No written contract says what a multimodal model must provide. The runtime finds multimodal behaviour through MultimodalModelMixin internals, so a model that does not inherit the mixin, such as a Modeling V2 target, has nothing to follow. Solution - Add SupportsMultimodal in tensorrt_llm/_torch/models/multimodal.py: capability flags, attention backends, language_model, embedding properties, the model-owned encoder cache, item scheduling, encoding and fusion. The module imports no model class. - is_multimodal_model() reads the supports_multimodal class marker. get_multimodal_data_device_paths() reads the optional multimodal_data_device_paths of any model. - MultimodalModelMixin sets the marker and implements the protocol. It exposes its cache as multimodal_encoder_cache and initialize_multimodal_encoder_cache(), and declares support_mm_disagg = False by default. - Align the encoder entry points of six models with the mixin's Sequence input, and document the contract in adding_new_model.md. - The executor does not change here, so a model still subclasses the mixin. The executor's mixin checks move to the protocol in a separate PR. Validation - CPU unit tests of the multimodal mixin pass, including the 4 new protocol tests. - Pre-commit hooks pass. Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
Problem Each multimodal runtime feature has its own flag. A model cannot say why it skips a feature, or limit a feature to some deployments. Solution - MultimodalFeature names the 12 runtime features. - multimodal_opt_outs (per class) and multimodal_feature_limits() (per deployment) list skipped features, each with a reason. - supports_feature() answers per feature: the class marker first, then opt-outs and limits, then today's flags (item scheduling, encoder cache, encoder-to-prefill handoff, encoder parallel modes, the LLM compile hook). Breakable CUDA graphs follow the language model class: its supports_multimodal_breakable_cuda_graph flag, or language_model_takes_rows. Names from open PRs are read with getattr defaults. - The executor does not call it yet, so today's flags stay the source of truth. Item scheduling and the encoder cache stay separate. Validation - New CPU tests cover the marker, each flag, the language model's graph flag, opt-outs, limits and both forms of the encoder parallel mode query. - Pre-commit hooks pass. Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
Problem A new multimodal model can skip a runtime feature and no test fails. Each feature is a separate opt-in, so new models often ship without them and get them months later. Solution - tiers.py lists the P0 and P1 features. A new model must support each P0 feature unless the multimodal owners approve an exception. A P1 opt-out needs a reason and a tracking ID in the model class. - models.py lists every multimodal architecture ID and every V2 target that its routes return. It does not rely on base classes. A text-only V2 target is only checked to refuse media and rows at admission, in validate_multimodal_request_data. - known_gaps.py records today's gaps. Features that no class flag shows are taken from the reviewed support matrix. test_contract.py holds the sizes, so the file can only shrink: the change that adds support deletes the entry, and new gaps go to APPROVED_EXCEPTIONS. - test_contract.py checks gaps, tiers and the class marker. test_processor_contract.py checks the processor hooks that item scheduling needs. An import test keeps the protocol module free of model and router modules. A broken registration fails one test, not the collection. - The multimodal onboarding skill and the Modeling V2 skills point to the same tiers, and a test keeps the skill table equal to tiers.py. - CBTS: a change to that skill runs this test. A Modeling V2 route, shared file or target that imports the multimodal contract runs the conformance suite. - CODEOWNERS lines for known_gaps.py and tiers.py. The owner team is still open. Not in this change, recorded as gaps: tiny model fixtures, the one- and two-GPU runtime feature suites, processor checks with real checkpoints, and the mRoPE, uint8 and media hash checks of open PRs. New models are not yet held to the shared base, fixtures, LLM breakable and piecewise CUDA graphs, or encoder DP/TP, because these are not on main yet. A model that opts out of one of them still needs an exception. Validation - The inventory finds 33 architecture IDs, 2 routed media targets and 14 routed text-only targets (2 refuse media at admission today). - CPU tests with the CPU CI marker on main 8ba86b8, CBTS tests included: 1501 pass on this branch and 1395 on main. The 1395 shared tests have the same outcome; the 106 new tests pass. - Pre-commit hooks pass. Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
yechank-nvidia
force-pushed
the
refactor/mm-protocol-20261023
branch
from
November 19, 2026 11:54
d8f213e to
829ff41
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Why. Every multimodal runtime feature (item scheduling, encoder cache, CUDA graphs, encoder parallelism, disaggregation) is a separate per-model opt-in that is off by default, and nothing fails when a new model skips one. Of the 6 multimodal wrappers added since August 2026, none runs the breakable CUDA graph with media, and only 2 had item scheduling and the encoder cache at launch (both because they subclass the Qwen3-VL base). The median time from onboarding a model to adding item scheduling is 359 days (encoder cache: 226 days). This PR writes down the contract and makes CI see the gaps.
What it adds. No behavior change: the executor keeps its existing checks, so a model still subclasses
MultimodalModelMixin.SupportsMultimodalintensorrt_llm/_torch/models/multimodal.py: what the runtime reads from a multimodal model, with thesupports_multimodalclass marker andis_multimodal_model().MultimodalModelMixinimplements it. The module imports no model class.MultimodalFeature(the 12 runtime features),multimodal_opt_outs(per class) andmultimodal_feature_limits()(per deployment). Each skip has a reason and a tracking ID.supports_feature(): one answer per feature, from the marker, opt-outs, limits and today's flags. The breakable CUDA graph follows the language model: its class flag, orlanguage_model_takes_rowswhen the language model places the rows itself. Names from open PRs are read withgetattrdefaults. Nothing calls it yet.tests/unittest/_torch/multimodal/conformance/:models.py: every multimodal architecture ID and every V2 target that its routes return (today 33 architecture IDs, 2 media targets, 14 text-only targets). It does not rely on base classes.validate_multimodal_request_data. 2 of the 14 do today; [TRTLLM-17780][feat] Run Modeling V2 vision-language models through the multimodal runtime (V1-style composite model + text-only guards) #22450 adds the rest.known_gaps.py: today's gaps. Features that no class flag shows come from the reviewed support matrix.test_contract.pyholds the sizes, so the file cannot grow: the change that adds support deletes the entry, and new gaps go toAPPROVED_EXCEPTIONS.tiers.py: the P0/P1 tiers below.tiers.py.known_gaps.pyandtiers.pyget their own lines. They name today's team; the multimodal owner team is still open.Tiers. P0: a new model must support it; skipping it needs an
APPROVED_EXCEPTIONSentry approved by the multimodal owners. P1: a new model may skip it with a reason and a tracking ID inmultimodal_opt_outs.disable_mm_encoder), item scheduling, encoder cache, chunked prefill with media, KV reuse with media hashes, LLM breakable CUDA graph with media, LLM piecewise CUDA graph and LLM-scoped compile, uint8 media, prefill-to-decode handoff, encoder DP eligibility, encoder attention backend declarationUntil their runtime parts are on main, new models are not held to the LLM breakable and piecewise CUDA graphs, encoder DP/TP, the shared base class and tiny fixtures (
STRUCTURAL_ONLYintiers.py). A model that opts out of one of them in its class still needs an exception. The graph rows follow the decoder: a decoder without a text graph mode needs no multimodal exception.Not in this PR.
isinstance(..., MultimodalModelMixin)checks to the protocol.known_gaps.pyentries the first makes stale (12 text-only admission keys, the qwen4_exp NVFP4 base gap and 3 of its feature gaps) and lowers the sizes intest_contract.py.Test Coverage
pytest -m cpu_only tests/unittest/_torch/multimodal/conformance(add-k <architecture>for one model).tests/unittest/_torch/multimodal/test_multimodal_mixin.py: the protocol, its defaults and eachsupports_featurerule.tests/unittest/scripts/cbts/test_docs.pyandtest_modeling_v2_rule.py: the new CI selection.-m cpu_only) of_torch/multimodal,_torch/pyexecutor/engine/test_multimodal.py,_torch/pyexecutor/test_perf_metrics_manager.py,_torch/modeling_v2/test_modeling_v2_contracts.pyandscripts/cbtson main42c12879: 1606 passed on this branch (829ff415a5) and 1500 on main. The 1500 shared tests have the same outcome, and the 106 new tests pass. An earlier, wider CPU run (-m "not gpu", withoutscripts/cbts) ond3dd0bfeand main8ba86b8balso gave the same outcome for all 398 shared tests.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.