Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
e35a86a
[None][feat] Mistral Tekken tokenizer: pickling, streaming control to…
yizhang-nv Nov 12, 2026
2be3698
[None][fix] KV-cache-aware router: load Mistral-native checkpoints as…
yizhang-nv Nov 12, 2026
6d6c9c8
[None][fix] Mistral-native serving: render chats with mistral-common,…
yizhang-nv Nov 12, 2026
ef1ca3b
[None][test] Mistral Tekken and Mistral-native tests: share the token…
yizhang-nv Nov 16, 2026
d150cd2
[None][fix] Mistral-native checkpoints: read the Tekken tokenizer on …
yizhang-nv Nov 16, 2026
e647b41
[None][fix] Mistral-native serving: refuse a chat_template in chat_te…
yizhang-nv Nov 16, 2026
bb67ff0
[None][chore] Mistral-native: leave out the reasoning-before-JSON gra…
yizhang-nv Nov 16, 2026
2c407a2
[None][fix] Mistral-native: mistral-common 1.13.0 on Python 3.11+, re…
yizhang-nv Nov 16, 2026
2bdfa0a
[None][fix] trtllm-bench: tune Mistral-native checkpoints from params…
yizhang-nv Nov 16, 2026
5aa25fe
[None][fix] trtllm-bench: report Mistral-native checkpoints in their …
yizhang-nv Nov 16, 2026
1e65927
[None][fix] trtllm-bench: read a Mistral-native model name from its s…
yizhang-nv Nov 16, 2026
5a68eb7
[None][perf] Mistral-native images: pass the loaded images to mistral…
yizhang-nv Nov 16, 2026
1729529
[None][test] Mistral-native: reasoning efforts are checked against th…
yizhang-nv Nov 16, 2026
5e4238d
[None][fix] Mistral-native guided decoding: llguidance covers the pad…
yizhang-nv Nov 16, 2026
a3d44e9
[None][fix] Mistral-native chats: pass history tool-call arguments to…
yizhang-nv Nov 16, 2026
e0c7ae8
[None][perf] KV-cache-aware router: one set of Mistral-native resourc…
yizhang-nv Nov 16, 2026
cae31f3
[None][chore] Mistral-native: one set lookup for image tokens; the st…
yizhang-nv Nov 16, 2026
1ddac55
[None][test] Mistral-native tests: check this change's behavior, not …
yizhang-nv Nov 17, 2026
d9c1246
[None][chore] MistralTokenizer: encoding_identity and grid_only_mistr…
yizhang-nv Nov 18, 2026
5596d7c
[None][perf] Mistral-native: the media loader hands mistral-common th…
yizhang-nv Nov 18, 2026
980df38
[None][feat] Mistral Large 4: carry feat/mistral-large-4 onto main
yizhang-nv Nov 4, 2026
f8435da
[None][chore] Mistral Large 4: regenerate the serve config schemas
yizhang-nv Nov 4, 2026
c2861d9
[None][test] Mistral Large 4: mark the RULER accuracy unit tests cpu_…
yizhang-nv Nov 4, 2026
fdcdfea
[None][fix] Mistral Large 4: index its multimodal model type in the l…
yizhang-nv Nov 4, 2026
beed5a4
[None][fix] Mistral Large 4: ask for a forced tool grammar only for r…
yizhang-nv Nov 4, 2026
cba6079
[None][test] Mistral Large 4: make the modeling_v2 attention tests a …
yizhang-nv Nov 4, 2026
22f1fea
[None][test] Mistral Large 4: follow main's dummy-request and KV mana…
yizhang-nv Nov 4, 2026
8005f50
[None][test] Mistral Large 4: align the checkpoint-backed engine test…
yizhang-nv Nov 4, 2026
22f19ac
[None][refactor] Mistral Large 4: move its catalog entries into the f…
yizhang-nv Nov 5, 2026
be9f135
[None][test] Mistral Large 4: expect main's bmm_out refusal for a 2-D…
yizhang-nv Nov 5, 2026
a6e1a0c
[None][test] Mistral Large 4: an over-ceiling top-k launch fails sile…
yizhang-nv Nov 5, 2026
9f8ce60
[None][refactor] DFlash/DSpark: name the draft-model methods the work…
yizhang-nv Nov 8, 2026
08c9a5e
[None][fix] Mistral Large 4: the DSpark drafter's logits are already …
yizhang-nv Nov 8, 2026
46ac5e1
[None][fix] Mistral Large 4: the standalone renderer skips the HF pro…
yizhang-nv Nov 8, 2026
8c14c3b
[None][doc] Mistral Large 4: list it among the supported models and i…
yizhang-nv Nov 8, 2026
f937ceb
[None][refactor] Mistral Large 4: name the targets by layout (tep, de…
yizhang-nv Nov 8, 2026
4ad6951
[None][test] Mistral Large 4: one accuracy gate per target, its deplo…
yizhang-nv Nov 8, 2026
a8abef8
[None][feat] Mistral Large 4: carry the VR200 (sm_107) targets from f…
yizhang-nv Nov 8, 2026
d77a3a1
[None][refactor] Mistral Large 4: load through checkpoint_format mistral
yizhang-nv Nov 12, 2026
b7b6fbc
[None][test] Mistral Large 4: GB300 perf-sanity cases from feat/mistr…
yizhang-nv Nov 12, 2026
b2c7718
[None][fix] Mistral Large 4: dummy loading skips the weight mapper of…
yizhang-nv Nov 12, 2026
b850149
[None][test] Mistral Large 4: one accuracy directory per target, as m…
yizhang-nv Nov 12, 2026
d4882b0
[None][test] Mistral Large 4: follow main's serve-test, environment-s…
yizhang-nv Nov 12, 2026
0ca0b54
[None][perf] Mistral Large 4: overlap the SM107 decode shared expert …
yizhang-nv Nov 17, 2026
7740ef3
[None][perf] Mistral Large 4: add a global split to sparse MLA decode…
yizhang-nv Nov 17, 2026
d4d7156
[None][perf] Mistral Large 4: use a direct BF16 GEMM for narrow SM107…
yizhang-nv Nov 17, 2026
f92fef9
[None][perf] Mistral Large 4: use a split radix Top-K for DSpark veri…
yizhang-nv Nov 17, 2026
1f5439b
[None][perf] Mistral Large 4: fuse FP8 quantization and GEMM for low-…
yizhang-nv Nov 17, 2026
9e621d7
[None][test] Mistral Large 4: run the SM107 decode ops' tests on GR100
yizhang-nv Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion ATTRIBUTIONS-Python.md
Original file line number Diff line number Diff line change
Expand Up @@ -15404,7 +15404,7 @@ CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
- `Homepage`: https://github.com/lark-parser/lark


## llguidance (0.7.29)
## llguidance (1.7.6)

### Licenses
License: `MIT`
Expand Down

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/source/deployment-guide/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,4 @@ The deployment guides below provide more detailed instructions for serving speci
deployment-guide-for-glm-5-on-trtllm.md
deployment-guide-for-glm-5.3-flash-on-trtllm.md
deployment-guide-for-minimax-m3-on-trtllm.md
deployment-guide-for-mistral-large-4-on-trtllm.md
5 changes: 5 additions & 0 deletions docs/source/models/supported-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ The following is a table of supported models for the PyTorch backend:
| `MiniCPMV4_6ForConditionalGeneration` [^14]| MiniCPM-V 4.6 | `openbmb/MiniCPM-V-4.6` |
| `MiniMaxM3SparseForConditionalGeneration` [^12]| MiniMax-M3 | `MiniMaxAI/MiniMax-M3` |
| `MistralForCausalLM` | Mistral | `mistralai/Mistral-7B-v0.1` |
| `MistralLarge4ForCausalLM` [^26] | Mistral Large 4 | `mistralai/Mistral-Large-4-1T-A52B-NVFP4` |
| `MixtralForCausalLM` | Mixtral | `mistralai/Mixtral-8x7B-v0.1` |
| `MllamaForConditionalGeneration` | Llama 3.2 | `meta-llama/Llama-3.2-11B-Vision` |
| `MuseSparkForConditionalGeneration` [^25] | Muse Spark Open | `meta-models/Muse-Spark-Open-816B-A42B-NVFP4` |
Expand Down Expand Up @@ -109,6 +110,7 @@ statuses for the same architecture in the two matrices.
| `Gemma4UnifiedForConditionalGeneration` | Yes | Yes | Yes | No | Yes | No | Yes | No | Yes | Yes | Yes |
| `Step3p7ForConditionalGeneration`| Yes | Yes | Yes | Untested | Untested | MTP | Yes | No | Yes | Yes | Yes |
| `MiniMaxM3SparseForConditionalGeneration` [^12] | Yes | Yes | Yes | Untested | Untested | EAGLE-3 (Linear), DSpark [^21] | Yes | No | N/A | Yes | Yes |
| `MistralLarge4ForCausalLM` [^26] | Yes | Yes | Yes | Yes [^27] | Yes | DSpark | Yes | Yes | N/A | Untested | Yes |

[^1]: Chunked Prefill for MLA can only be enabled on SM90/SM100/SM103/SM107/SM120.
[^2]: KV cache reuse for MLA can only be enabled on SM90/SM100/SM103/SM107/SM120/SM121 and in BF16/FP8 KV cache dtype.
Expand All @@ -131,6 +133,8 @@ statuses for the same architecture in the two matrices.
[^23]: Text-only decision model. `trtllm-serve embeddings` detects the native checkpoint and selects the FlashInfer attention backend automatically. Python `LLM.encode` callers must set `model_kwargs.architectures` to `[PplxDeciderForDecision]`. The embedding vector contains the 255 raw decision logits; apply the checkpoint's calibration and candidate normalization in the caller.
[^24]: DeepSeek-V4.1 uses CSA2 sparse attention; see [DeepSeek-V4.1 CSA2](../features/sparse-attention.md#deepseek-v41-csa2). Checkpoints with candidate-consumer layers, including the released one, run only on SM100-family GPUs with a DeepGEMM build that includes the sparse MQA-logits kernels; otherwise model construction fails. Checkpoints without such layers can also use the SM90 and SM120/SM121 attention routes. The Engram tables live in pinned host memory, placed on a GPU's host NUMA node when the GPUs report one. When that node lacks free memory, model loading evicts the node's page cache, which every process on the node loses, and retries. Set `TRTLLM_ENGRAM_EVICT_PAGE_CACHE=0` to skip the eviction; such a node's tables then fall back to host memory that can span NUMA nodes, which can slow lookups.
[^25]: Muse Spark Open is supported on NVIDIA Blackwell GPUs (`SM100` family); the validated deployment is TP4/EP4 on one 4-GPU GB300 NVL node — see the [Muse Spark Open deployment guide](../deployment-guide/deployment-guide-for-musespark-open-on-trtllm.md). Supports text, image, video, and audio inputs. On `SM103`, chunked prefill and KV cache reuse require the FP8 KV cache dtype (head-dimension-64 paged-context FMHA constraint); disaggregated serving runs only on the NIXL cache-transceiver backend with the Python transceiver runtime; some disaggregated configurations additionally require `TLLM_FMHA_LIBS=-fallback` in the server environment and, on GB300 (aarch64), `UCX_TLS=cuda_copy,cuda_ipc,sm,self,tcp`. Speculative decoding uses a DFlash2 drafter ([`nvidia/Muse-Spark-Open-816B-A42B-DFlash2`](https://huggingface.co/nvidia/Muse-Spark-Open-816B-A42B-DFlash2)).
[^26]: Mistral Large 4 runs only on NVIDIA GB300 (`SM103`) from the NVFP4 checkpoint in the Mistral format (`checkpoint_format: mistral`); the FP8 checkpoint and the Hugging Face format are not supported. The architecture name is the one TensorRT LLM assigns when it reads the checkpoint's `params.json`. It is served by three modeling_v2 targets, tensor-parallel attention with expert-parallel experts, attention data parallelism with expert-parallel experts, and tensor parallelism, each on 4 GPUs or on 8 GPUs over two nodes, with contexts of up to 1,048,576 tokens. Speculative decoding uses the DSpark draft in the checkpoint (3 draft tokens). See the [Mistral Large 4 deployment guide](../deployment-guide/deployment-guide-for-mistral-large-4-on-trtllm.md).
[^27]: Validated for one TEP4 context server and one TEP4 generation server at a 32K-token context, without speculative decoding.

# Encoder-Decoder Feature Support Matrix (PyTorch Backend)

Expand Down Expand Up @@ -181,6 +185,7 @@ not used to mark a feature as supported here.
| `Qwen3_5ForConditionalGeneration` | Yes | Yes | Untested | Yes | No | Untested | Yes | L + I + V |
| `Qwen3_5MoeForConditionalGeneration` | Yes | Yes | Untested | Yes | No | Untested | Yes | L + I + V |
| `Qwen4ExpForConditionalGeneration` | Untested | Untested | Untested | Yes | Untested | Yes | No | L + I |
| `MistralLarge4ForCausalLM` [^26] | Yes | Yes | Yes | Yes | Yes | Untested | Untested | L + I |

Note:
- L: Language
Expand Down
15 changes: 15 additions & 0 deletions examples/serve/compatibility/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,21 @@ This directory contains individual, self-contained examples demonstrating Tensor
trtllm-serve Qwen/Qwen3-8B --reasoning_parser "qwen3" --tool_parser "qwen3"
```

Native `mistral_large_4` required and named tool choices need the `llguidance`
guided-decoding backend. Override an xgrammar serving configuration with:

```bash
trtllm-serve /path/to/checkpoint --config serve.yaml \
--tool_parser mistral_large_4 --reasoning_parser mistral_large_4 \
--set guided_decoding_backend=llguidance
```

Required tool choices allow parallel calls unless the request explicitly sets
`parallel_tool_calls` to `false`. Named choices constrain generation to one
call, including when `parallel_tool_calls` is `true`, matching the single-call
forced-choice post-processing. Native required and named choices return HTTP
400 when the server uses xgrammar.


## Running Examples

Expand Down
8 changes: 5 additions & 3 deletions jenkins/L0_Test.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -7884,7 +7884,7 @@ def launchTestJobs(pipeline, testFilter, globalVars)
"GB300-4_GPUs-PyTorch-PerfSanity-KeyModels-Post-Merge-{split}_of_2",
"auto:gb300-x4",
"l0_gb300_multi_gpus_perf_sanity_key_models",
2,
4,
4,
1, true
)
Expand Down Expand Up @@ -7986,8 +7986,10 @@ def launchTestJobs(pipeline, testFilter, globalVars)
"GB300-16_GPUs-4_Nodes-PyTorch-PerfSanity-Node4-GPU16-Post-Merge-1_of_2": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_node4_gpu16", 1, 1, 16, 4],
// gen_only_no_context: gen1 (8 nodes, 32 GPUs), no ctx fleet = 32 GPUs
"GB300-32_GPUs-8_Nodes-PyTorch-PerfSanity-GEN1-NODE8-GPU32-Post-Merge-3_of_4": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_gen1_node8_gpu32", 1, 1, 32, 8],
// GLM-5.3-Flash FP8 8k1k con16: ctx1 (1 node, 4 GPUs) + gen1 tep4 (1 node, 4 GPUs) = 8 GPUs
"GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-KeyModels-Post-Merge-1_of_2": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4_key_models", 1, 1, 8, 2],
// GLM-5.3-Flash FP8 8k1k con16 and Mistral Large 4 NVFP4 8k1k con64/256/1024:
// ctx1 (1 node, 4 GPUs) + gen1 (1 node, 4 GPUs) = 8 GPUs, one split per build slot
"GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-KeyModels-Post-Merge-1_of_2": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4_key_models", 1, 2, 8, 2],
"GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-KeyModels-Post-Merge-2_of_2": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4_key_models", 2, 2, 8, 2],
// Nemotron-Ultra-V3 8k64k con1: ctx1 (1 node, 4 GPUs) + gen1 tep4 (1 node, 4 GPUs) = 8 GPUs
"GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3_of_4": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4", 1, 1, 8, 2],
]
Expand Down
5 changes: 3 additions & 2 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ einops
fla-core==0.5.2 # the Kimi Linear / Kimi K3 KDA layers import fla.modules and fla.ops when they load
flashinfer-python==0.7.3
xgrammar==0.2.8
llguidance==0.7.29
llguidance==1.7.6
jsonschema
backoff
nvtx
Expand Down Expand Up @@ -106,7 +106,8 @@ mcp>=1.30.0,<2.0.0 # 2.0.0 removed mcp.server.fastmcp; scaffolding still uses th
apache-tvm-ffi==0.1.13.post2 # required by the CUTLASS DSL TVM-FFI provider
torch-c-dlpack-ext==0.1.3 # used for reduce nvidia-cutlass-dsl host overhead, optional package for improved torch tensor calling perf
flash-attn-4==4.0.0b19
mistral-common>=1.10.0
mistral-common[opencv]>=1.13.0; python_version >= "3.11"
mistral-common[opencv]>=1.12.0; python_version < "3.11" # 1.13.0 requires Python 3.11+
torchao>=0.14.1,<0.16.0
cuda-core
llist
Expand Down
4 changes: 2 additions & 2 deletions security_scanning/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ dependencies = [
"fla-core (==0.5.2)",
"flashinfer-python (==0.7.3)",
"xgrammar (==0.2.8)",
"llguidance (==0.7.29)",
"llguidance (==1.7.6)",
"jsonschema (>=4.26.0,<5.0.0)",
"backoff (>=2.2.1,<3.0.0)",
"nvtx (>=0.2.17,<0.3.0)",
Expand All @@ -86,7 +86,7 @@ dependencies = [
"mcp (>=1.30.0,<2.0.0)",
"torch-c-dlpack-ext (==0.1.3)",
"flash-attn-4 (==4.0.0b19)",
"mistral-common (>=1.10.0)",
"mistral-common[opencv] (>=1.13.0) ; python_version >= \"3.11\"",
"torchao (>=0.14.1,<0.16.0)",
"cuda-core (>=1.3.1,<2.0.0)",
"llist (>=0.7.1,<0.8.0)",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -674,10 +674,13 @@ def __init__(
)
primary_converters = [
self.impl.get_page_index_converter(local_layer_idx, Role.KEY)
if self.has_buffer(local_layer_idx, Role.KEY)
else None
for local_layer_idx in range(self.num_local_layers)
]
self._primary_pool_page_index_params = [
(int(converter.scale), int(converter.layer_offset)) for converter in primary_converters
(int(converter.scale), int(converter.layer_offset)) if converter is not None else None
for converter in primary_converters
]
local_indexer_mask = self.indexer_k_cache_local_layer_mask
full_indexer_local_layers = [
Expand Down Expand Up @@ -719,8 +722,11 @@ def indexer_k_cache_local_layer_mask(self) -> list[bool]:
# The base wrapper may remove empty layers while building the cache.
return [self._indexer_k_cache_layer_mask[layer_idx] for layer_idx in self.pp_layers]

def get_primary_pool_page_index_params(self, local_layer_idx: int) -> Tuple[int, int]:
"""Return the formal V2 page scale and layer offset for sparse MLA."""
def get_primary_pool_page_index_params(self, local_layer_idx: int) -> Optional[Tuple[int, int]]:
"""Return the formal V2 page scale and layer offset for sparse MLA.

None for a layer that owns no latent KEY buffer.
"""
return self._primary_pool_page_index_params[local_layer_idx]

def _get_buffer_roles_for_layer(self, local_layer_idx: int) -> List[DataRole]:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

from .cache_manager import MistralLarge4CacheManager

__all__ = ["MistralLarge4CacheManager"]
Loading
Loading