Repository navigation
[https://nvbugs/7090784][fix] Fix MiniMax M3 fullgraph context warmup #22479
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
peihu-nv
wants to merge
10
commits into
NVIDIA:main
Choose a base branch
from
peihu-nv:fix/nvbug-7090784-compile-reporting
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,373
−329
Open
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
ec9f7e9
[https://nvbugs/7090784][fix] Keep MiniMax M3 dispatch reporting outs…
peihu-nv 4cecca3
[https://nvbugs/7090784][fix] Keep prefill host state outside compile…
peihu-nv c22e5f8
[https://nvbugs/7090784][fix] Keep live M3 attention outside captured…
peihu-nv e160031
[https://nvbugs/7090784][test] Check live rows at the prefill produce…
peihu-nv 8db1a86
[https://nvbugs/7090784][test] Cover MiniMax M3 TP4 prefill compile r…
2ez4bz a4555f0
address comments
2ez4bz 201b5f3
[https://nvbugs/7090784][test] Add shared MiniMax M3 fixtures
2ez4bz 2fc36b1
address bot comments once more...
2ez4bz 7c52d39
[https://nvbugs/7090784][fix] Keep compiled M3 prefill on one graph p…
2ez4bz 4b6dd71
[https://nvbugs/7090784][test] Trim MiniMax M3 compiled prefill tests
2ez4bz File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
240 changes: 135 additions & 105 deletions
240
tensorrt_llm/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/modeling.py
Large diffs are not rendered by default.
Oops, something went wrong.
207 changes: 143 additions & 64 deletions
207
tensorrt_llm/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp4/modeling.py
Large diffs are not rendered by default.
Oops, something went wrong.
237 changes: 158 additions & 79 deletions
237
tensorrt_llm/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp8/modeling.py
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
133 changes: 133 additions & 0 deletions
133
tensorrt_llm/_torch/modeling_v2/models/minimax_m3/prefill_attention.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| """Keep request-dependent M3 attention outside captured prefill segments.""" | ||
|
|
||
| from collections.abc import Callable | ||
| from dataclasses import dataclass | ||
| from functools import partial | ||
| from weakref import ReferenceType, ref | ||
|
|
||
| import torch | ||
| from torch import nn | ||
|
|
||
| from tensorrt_llm._torch.utils import get_model_extra_attrs | ||
|
|
||
|
|
||
| @dataclass | ||
| class _PrefillCore: | ||
| core: ReferenceType[nn.Module] | ||
| producer: Callable[..., tuple[torch.Tensor, torch.Tensor] | None] | ||
| attention: Callable[..., None] | ||
| indexer: Callable[..., torch.Tensor] | ||
|
|
||
|
|
||
| _CORES: dict[str, _PrefillCore] = {} | ||
|
|
||
|
|
||
| def _forget_core(key: str, reference: ReferenceType[nn.Module]) -> None: | ||
| _CORES.pop(key, None) | ||
|
|
||
|
|
||
| def register_prefill_core( | ||
| core: nn.Module, | ||
| producer: Callable[..., tuple[torch.Tensor, torch.Tensor] | None], | ||
| attention: Callable[..., None], | ||
| indexer: Callable[..., torch.Tensor], | ||
| ) -> str: | ||
| """Register ``core``'s live-row attention callbacks and return the key the boundary takes. | ||
|
|
||
| The registry holds ``core`` weakly, so releasing the model drops its entry. | ||
| """ | ||
| key = f"m3_prefill_{id(core)}" | ||
| _CORES[key] = _PrefillCore(ref(core, partial(_forget_core, key)), producer, attention, indexer) | ||
| return key | ||
|
|
||
|
|
||
| def run_prefill_attention( | ||
| packed: torch.Tensor | None, | ||
| position_ids: torch.Tensor | None, | ||
| q: torch.Tensor | None, | ||
| k: torch.Tensor | None, | ||
| v: torch.Tensor | None, | ||
| idx_qk: torch.Tensor | None, | ||
| output: torch.Tensor, | ||
| core_key: str, | ||
| layer: int, | ||
| ) -> None: | ||
| """Run one prefill layer's attention over the live rows of the current attention metadata. | ||
|
|
||
| A sparse layer with the five-way projection passes ``packed``: the horizontal producer writes K, V and index-K | ||
| to the paged caches and returns Q and index-Q. An unfused sparse layer passes ``idx_qk``, whose index-K is | ||
| written before attention. A dense layer passes only ``q``, ``k`` and ``v``. Pad rows of ``output`` are left as | ||
| allocated. | ||
|
|
||
| Breakable prefill graphs call this function directly at every replay. Compiled prefill calls it through | ||
| ``prefill_attention``, a piecewise boundary op. | ||
| """ | ||
| entry = _CORES[core_key] | ||
| core = entry.core() | ||
| if core is None: | ||
| raise RuntimeError("MiniMax-M3 prefill core is no longer available") | ||
| metadata = get_model_extra_attrs()["attention_metadata"]() | ||
| live = int(metadata.num_tokens) | ||
| if position_ids is not None: | ||
| position_ids = position_ids[..., :live] | ||
| idx_q = None | ||
| if packed is not None: | ||
| horizontal = entry.producer( | ||
| core, | ||
| layer, | ||
| packed[:live], | ||
| position_ids, | ||
| metadata.kv_cache_manager, | ||
| metadata.msa_idx_k_cache, | ||
| metadata.msa_out_cache_loc, | ||
| ) | ||
| if horizontal is None: | ||
| raise RuntimeError( | ||
| f"MiniMax-M3 sparse layer {layer}: no compatible horizontal producer" | ||
| ) | ||
| q, idx_q = horizontal | ||
| k = v = None | ||
| elif idx_qk is not None: | ||
| # Insert index-K where attention reads it, using this step's live cache slots. | ||
| idx_q = entry.indexer( | ||
| core, | ||
| layer, | ||
| idx_qk[:live], | ||
| position_ids, | ||
| metadata.msa_idx_k_cache, | ||
| metadata.msa_out_cache_loc, | ||
| ) | ||
| entry.attention(core, layer, q, k, v, idx_q, metadata, output) | ||
|
|
||
|
|
||
| @torch.library.custom_op("trtllm::modeling_v2_m3_prefill_attention", mutates_args=("output",)) | ||
| def prefill_attention( | ||
| packed: torch.Tensor | None, | ||
| position_ids: torch.Tensor | None, | ||
| q: torch.Tensor | None, | ||
| k: torch.Tensor | None, | ||
| v: torch.Tensor | None, | ||
| idx_qk: torch.Tensor | None, | ||
| output: torch.Tensor, | ||
| core_key: str, | ||
| layer: int, | ||
| ) -> None: | ||
| """``run_prefill_attention`` as an op the piecewise optimizer splits on (it is not captured).""" | ||
| run_prefill_attention(packed, position_ids, q, k, v, idx_qk, output, core_key, layer) | ||
|
|
||
|
|
||
| @prefill_attention.register_fake | ||
| def _prefill_attention_fake( | ||
| packed: torch.Tensor | None, | ||
| position_ids: torch.Tensor | None, | ||
| q: torch.Tensor | None, | ||
| k: torch.Tensor | None, | ||
| v: torch.Tensor | None, | ||
| idx_qk: torch.Tensor | None, | ||
| output: torch.Tensor, | ||
| core_key: str, | ||
| layer: int, | ||
| ) -> None: | ||
| pass |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
68 changes: 68 additions & 0 deletions
68
tests/unittest/_torch/modeling_v2/models/minimax_m3/conftest.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| import pytest | ||
| import torch | ||
| from transformers import PretrainedConfig | ||
|
|
||
| from tensorrt_llm._torch.model_config import ModelConfig | ||
| from tensorrt_llm.llmapi.llm_args import MiniMaxM3SparseAttentionConfig | ||
| from tensorrt_llm.mapping import Mapping | ||
| from tensorrt_llm.models.modeling_utils import QuantConfig | ||
| from tensorrt_llm.quantization.mode import QuantAlgo | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| def make_m3_config(): | ||
| def make_config( | ||
| tp: int, | ||
| *, | ||
| moe_layers: tuple[int, ...] = (0, 1), | ||
| sparse_layers: tuple[int, ...] = (0, 1), | ||
| head_dim: int = 128, | ||
| fused_projection: bool = False, | ||
| ) -> ModelConfig[PretrainedConfig]: | ||
| return ModelConfig( | ||
| pretrained_config=PretrainedConfig( | ||
| dtype=torch.bfloat16, | ||
| vocab_size=128, | ||
| hidden_size=128, | ||
| num_hidden_layers=len(moe_layers), | ||
| num_attention_heads=tp, | ||
| num_key_value_heads=tp, | ||
| head_dim=head_dim, | ||
| max_position_embeddings=32, | ||
| dense_intermediate_size=256, | ||
| shared_intermediate_size=256, | ||
| intermediate_size=256, | ||
| num_local_experts=8, | ||
| num_experts_per_tok=2, | ||
| n_shared_experts=1, | ||
| rms_norm_eps=1e-6, | ||
| moe_layer_freq=moe_layers, | ||
| sparse_attention_config={ | ||
| "sparse_attention_freq": sparse_layers, | ||
| "sparse_num_index_heads": tp, | ||
| "sparse_index_dim": 128, | ||
| }, | ||
| ), | ||
| mapping=Mapping(world_size=tp, tp_size=tp), | ||
| quant_config=QuantConfig(kv_cache_quant_algo=QuantAlgo.FP8), | ||
| sparse_attention_config=MiniMaxM3SparseAttentionConfig( | ||
| implementation="msa", | ||
| indexer_kv_dtype="fp8", | ||
| sparse_num_index_heads=tp, | ||
| fuse_qkv_index_projection=fused_projection, | ||
| ), | ||
| allreduce_strategy="NCCL", | ||
| max_num_tokens=32, | ||
| ) | ||
|
|
||
| return make_config | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| def make_cuda_m3_config(make_m3_config): | ||
| if not torch.cuda.is_available(): | ||
| pytest.skip("MiniMax-M3 constructors allocate CUDA resources") | ||
| return make_m3_config |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.