Skip to content

[None][feat] Merge back Mistral Large 4 support from feat/mistral-large-4 - #21262

Draft
yizhang-nv wants to merge 20 commits into
mainfrom
feat/mistral-large-4
Draft

yizhang-nv wants to merge 20 commits into
mainfrom
feat/mistral-large-4

Conversation

@yizhang-nv

Copy link
Copy Markdown
Member

Description

This PR merges the Mistral Large 4 support on the feat/mistral-large-4 release branch back into main.

feat/mistral-large-4 is the branch published for the Mistral Large 4 launch, and Mistral's launch announcement links to it. It holds one squashed commit, fece87edc4 ("[None][feat] Add Mistral Large 4 support"), on top of main at 59eab99c89. Merging it back puts Mistral Large 4 support into main and into regular TensorRT LLM releases.

Note

Placeholder, not ready for review. This draft tracks the merge-back. The branch has not been rebased onto current main yet. The PR will be rebased, its scope finalized and CI run before reviewers are requested.

Summary of changes

Model support (modeling_v2)

  • Adds three modeling_v2 targets for the NVFP4 checkpoint mistralai/Mistral-Large-4-1T-A52B-NVFP4 on GB300 (sm_103): mistral_large_4_nvfp4__sm_103__tep4, __dep4 and __tp4. The two-node, 8-GPU deployments (TEP8, DEP8, TP8) use the same targets.
  • Supports contexts of up to 1,048,576 tokens with chunked prefill, on an FP8 latent (fp8_ds_mla) KV cache with a dedicated cache manager (tensorrt_llm/_torch/attention/backends/sparse/mistral_large4/).
  • Supports image input and DSpark speculative decoding with the draft embedded in the checkpoint.
  • Adds modeling_v2 catalog ops for sparse MLA attention (FlashMLA sparse prefill/decode, CuTe DSL sparse decode) and the sparse top-k indexer (FP8 MQA logits, top-k prefill/decode, indexer K-cache update). Also adds SSMax query scaling, the final-logit soft-cap, a fused allreduce + RMSNorm, FP8 block-scale GEMM and quantization, MoE routing and the Pixtral 2D RoPE. Existing catalog entries get the shapes these targets use.

Checkpoint loading

  • Adds checkpoint_format: mistral_large_4. With it, the model configuration, weights, tokenizer (tekken.json through mistral-common), chat rendering and image preprocessing all come from the Mistral-format files; the Hugging Face files are not read. The FP8 checkpoint is refused at load time.
  • Moves DSpark draft resolution out of llm_args.py into the checkpoint adapters (tensorrt_llm/_torch/models/checkpoints/dspark_config.py), which also handle a draft embedded in a Mistral-format checkpoint.

Serving

  • Adds the mistral_large_4 tool parser, the mistral_large_4 reasoning parser and a serving extension. The extension renders chats with mistral-common and takes the reasoning mode and response format from mistral-common's model settings.
  • Supports structured outputs with the xgrammar and llguidance backends over the Tekken vocabulary. The guided-decoding encoded_vocab now accepts raw token bytes.
  • Chat requests accept OpenAI's store=false, and history tool calls whose arguments are not a JSON object.
  • Validates disaggregated serving with one TEP4 context server and one TEP4 generation server.

Runtime and tools

  • Fixes the guided decoder's interaction with eager forwards and the decoder warmup when the guided decoder is capturable.
  • Patches FlashMLA to release the GIL in its pybind launch wrappers (3rdparty/patches/flashmla_release_gil.patch).
  • trtllm-bench reads the checkpoint in the format named by extra_llm_api_options.
  • lm-eval tasks are scored on the response, not on the reasoning section.

Dependencies

  • llguidance 0.7.29 → 1.7.6, because 0.7.29 cannot build the Tekken token trie.
  • mistral-common>=1.10.0 → mistral-common[opencv]>=1.13.0. 1.13.0 knows the checkpoint's reasoning efforts, and the opencv extra is needed for image input.

Documentation

  • Adds the deployment guide docs/source/deployment-guide/deployment-guide-for-mistral-large-4-on-trtllm.md, which lists the supported configurations, features and known limitations.

Test Coverage

  • Unit tests:
    • the new and extended modeling_v2 catalog ops (tests/unittest/_torch/modeling_v2/);
    • per-target engine configurations and weight manifests (cpu_only);
    • the sparse KV cache manager and layout (tests/unittest/_torch/attention/sparse/mistral_large4/);
    • the tokenizer factory, tool and reasoning parsers, chat utilities and the serving extension;
    • disaggregated KV transfer (tests/unittest/disaggregated/test_mistral_large4_kv_transfer.py).
  • Integration accuracy tests in tests/integration/defs/accuracy/modeling_v2/test_mistral_large4.py:
    • GSM8K for every target;
    • DSpark accuracy (GSM8K, RULER-lite, acceptance length);
    • llguidance structured outputs.
  • Disaggregated serving accuracy in tests/integration/defs/accuracy/test_mistral_large4_disaggregated.py.
  • Test lists: l0_cpu.yml, l0_gb300.yml, l0_gb300_multi_gpus.yml and l0_gb300_multi_nodes_node2_gpu8.yml.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

🤖 Generated with Claude Code

Add Mistral Large 4 to the PyTorch backend as modeling_v2 targets, with
serving support and tests.

Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
sharabeshj-nvda and others added 9 commits October 30, 2026 19:19
Signed-off-by: Sharabesh Jayaraman <338298191+sharabeshj-nvda@users.noreply.github.com>
(cherry picked from commit eb6fc51)
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: Yuanjing Xue <197832395+yuanjingx87@users.noreply.github.com>
Co-authored-by: Yanchao Lu <yanchaol@nvidia.com>
Co-authored-by: tburt-nv <195370667+tburt-nv@users.noreply.github.com>
(cherry picked from commit 30a9367)
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
…ror (#21068)

Signed-off-by: Sharabesh Jayaraman <338298191+sharabeshj-nvda@users.noreply.github.com>
(cherry picked from commit 9c5e8e5)
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
…21336)

Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
(cherry picked from commit ee7c59a)
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>
(cherry picked from commit 00c184e)
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
…el selection (#20596)

Signed-off-by: Yiteng Niu <6831097+niukuo@users.noreply.github.com>
(cherry picked from commit 0084e98)
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
mistral-common 1.13.0 requires Python 3.11+, so the PY310 package sanity
stage could not install requirements and never built the cp310 wheel.
Fall back to mistral-common>=1.10.0 on Python 3.10.

Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
…eel (#21425)

Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
yizhang-nv and others added 4 commits November 4, 2026 15:07
…d mapper (#21548)

The mistral_large_4 checkpoint format registered the generic HF weight
loader and mapper, and HfWeightLoader._requires_lazy_safetensors sniffed
every checkpoint directory for a Mistral-native params.json to force lazy
SafeTensors for Mistral Large 4.

Move both into checkpoints/mistral/:
- MistralLarge4WeightLoader subclasses HfWeightLoader, always keeps the
  checkpoint mmapped, and is the default weight loader of
  MistralLarge4CheckpointLoader.
- MistralLarge4WeightMapper subclasses HfWeightMapper and renames
  nothing, as before.

hf/weight_loader.py and hf/weight_mapper.py return to their state before
Mistral Large 4 support was added.

Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
…of removed paths (#21616)

Remove defensive code that Mistral Large 4 support added, and code that
only served paths removed earlier (the FP8 target and mistral-common
releases before 1.13.0):

- Require mistral-common 1.13.0 with one version check where the native
  config loader recognizes a Mistral Large 4 params.json, and drop the
  tokenizer and JsonSchema fallbacks for older releases.
- Drop the FP8 target's leftovers: the input processor's serves-images
  check, the NVFP4 condition in routing.serves_images and the refusal of
  a params.json quantization block (routing rejects it as quantization
  that no target fits).
- Check each chat request rule once: the native renderer no longer
  rejects chat_template, and the serving extension no longer prechecks
  reasoning-effort names that mistral-common validates when rendering.
- Drop checks that routing or the callers already guarantee: the cache
  manager's structural checks become one assert; the sharing-length and
  DSpark layer-order checks, the recurrent DSpark head raise, the
  attention-DP payloads without spec-decoding fields, the "owns no KEY
  buffer" raises, the native image count invariants and lm_eval's token
  id validation go.
- State dsa.fp8_kv_cache as the ModelOpt kv_cache_quant_algo in the
  native config adapter, so ModelConfig and ModelLoader need no Mistral
  Large 4 branch.

Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
chenfeiz0326 added a commit to chenfeiz0326/TensorRT-LLM that referenced this pull request Nov 11, 2026
… models

Mistral Large 4 perf-sanity cases run once every 2 post-merge builds,
like the other key models. Its cases (NVIDIA#22056) reach main with NVIDIA#21262 and
belong in the _key_models lists.

Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>
…ith the routed experts (#22253)

Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
…de on SM107 (#22252)

Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
… projections on SM107 (#22251)

Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
…ication on SM107 (#22249)

Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
… decode on SM107 (#22250)

Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
Signed-off-by: tayuan <51468697+yt0428@users.noreply.github.com>
from tensorrt_llm._torch.disaggregation.native.utils import get_local_ip

with socket.socket() as sock:
sock.bind(("", 0))
@ruocheng-nv

Copy link
Copy Markdown
Collaborator

Hi, heads-up: #22286 has merged, so ModelingV2 now lives at tensorrt_llm/_torch/modeling_v2 (module tensorrt_llm._torch.modeling_v2), and the old _torch/_experimental/modeling_v2 path no longer exists.

Since this PR touches modeling_v2, please rebase it onto the latest main. You can use these migration commands to move any files you added under the old directory and update any remaining references to the old path.

…s own decode selector (#22467)

Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants